Category: Exchange

These are related to Exchange Server and topics along those lines.

Author Lance LingerfeltPosted on

Exchange 2010 Extended Support will end on October 13th, 2020

I wanted to pass this announcement along to everyone so that they are aware of the support ending for Exchange 2010. I personally have noticed a large number of Exchange 2010 environments starting to show age as the newer Outlook clients are having performance issues with Exchange 2010. If your team has not planned an upgrade to Exchange 2016 (you cannot upgrade directly from Exchange 2010 to 2019), I would advise that your team do so very soon. Exchange 2010 has been a great product for many years, but it is finally time for it to retire and allow the next generation of Messaging Services take the stage.

Formal Announcement:

Exchange 2010 End of Support extended to October 2020

Announced today, and in alignment with Office 2010 and SharePoint 2010, and after investigating and analyzing the deployment state of an extensive number of Exchange customers, Microsoft has decided to move Extended Support date for Exchange Server 2010 from January 14th 2020 to October 13th 2020.
After October 13th 2020, Microsoft will no longer provide technical support for problems that may occur with Exchange 2010 including:

– bug fixes for issues that are discovered and that may impact the stability and usability of the server
– security fixes for vulnerabilities that are discovered and that may make the server vulnerable to security breaches
– and time zone updates

Customer installations of Exchange 2010 will, of course, continue to run after this date; however, due to the changes and potential end of support risks, Microsoft strongly recommends customers migrate from Exchange 2010 as soon as possible.

FAQ’s

  • Can customers upgrade directly to Exchange 2019?
    Customers cannot upgrade directly from Exchange 2010 on-premises to Exchange Server 2019. They may upgrade to Exchange 2013 or 2016 directly from Exchange 2010 and we of course recommend Exchange 2016.
  • Since Exchange 2010 runs on Server 2008 and 2008R2, are those operating systems still supported?
    On January 14, 2020, support for Windows Server 2008 and 2008 R2 will end. That means the end of regular security updates for these Windows customers. Since Exchange Server 2010 runs on top of Windows Server 2008 and Windows Server 2008 R2, it’s important for customers to consider how they will obtain security updates for the underlying operating system. Extended Security Updates for Server 2008 and 2008 R2 are now available for purchase and can be ordered from Microsoft or a Microsoft licensing partner. The delivery of Extended Security Updates (ESU) will begin after the End of Support dates, if and when available. 
  • Does Microsoft support Exchange 2010 on any other Server versions?
    Exchange Server 2010 SP3, with Update Rollup 26 or higher, installed on Windows Server 2012 R2 is supported until October 13, 2020.
  • That didn’t quite answer my question. If a customer calls between January 14 and October 13 2020, and is running Exchange 2010 on Server 2008 or 2008 R2, and does not have an ESU for Windows, can they still be assisted?
    Yes. Per the Lifecycle FAQ.
    If I am running a Microsoft product that is currently supported under the Lifecycle Policy, but my operating system is no longer supported, can I still receive support?
    If the problem is specific to the Microsoft product and it is within the Lifecycle Policy, Microsoft will provide support.
    If the problem is a result of the combination of the operating system and the Microsoft product, the problem will not be supported.
    More simply::
    Exchange 2010 on Server 2008 or 2008 R2: Starting January 14, 2020, provide support until a proven issue is found with the OS. This ends in October 2020.
    Exchange 2010 SP3 RU26+ on Server 2012 R2: We support regardless, but Exchange support still ends in October 2020.
  • Will Microsoft be offering Extended Support Updates (ESU’s) for purchase for Exchange 2010 customers?
    No.
  • What resources are available for customers? 
    – An upcoming Exchange Team blog post, titled “Exchange On-Premises Best Practices for Migrations from 2010 to 2016,” will provide great technical guidance for customers and support agents with their on-premises migrations.  
    – If migrating to Office 365 and Exchange Online, customers may be eligible to use the free Microsoft FastTrack service. FastTrack provides best practices, tools, and resources to make migration to Office 365 and Exchange Online as seamless as possible.
    – For customers that run into any problems during their migration to Office 365 and are not eligible for FastTrack, or if migrating to a newer version of Exchange Server, customers can of course utilize Support or the Exchange Technical Community.
    – Customers may also choose to engage a partner to help.  Microsoft has a great number of partners with deep skills in Exchange, and you can browse a list of Exchange partners at     https://www.microsoft.com/en-us/solution-providers/home.

HAPPY UPGRADING!
CONTACT ME FOR QUESTIONS CONCERNING UPGRADING YOUR EXCHANGE ENVIRONMENT!

Author Lance LingerfeltPosted on

Exchange Server Client Access URL Configuration Script

In my career, I have to be able to be efficient as most of my projects are on a time crunch schedule. Being able to quickly configure Exchange when setting up a server environment is crucial to the success of the project.

While still honing my skills in PowerShell, I was attempting to create my own script to help configure all of the Virtual Directories in one shot rather than go to each setting and configure them manually. It did not go very well, so as I do, I research and find great professionals that do great work in scripting so that I may learn from them.

In doing so, I found Paul Cunningham’s script that performs this. I took the following script and modified it to add the PowerShell Virtual Directory to it as I like to configure that as well.

***YOU CAN REM THE LINES OUT SHOULD YOU NOT WANT TO CONFIGURE THAT DIRECTORY***

Here is my version of the script:

NOTES:

  • PowerShell script to configure the Client Access server URLs for Microsoft Exchange Server 2013/2016. All Client Access server URLs will be set to the same namespace.
  • If you are using separate namespaces for each CAS service this script will not handle that.
  • The script sets Outlook Anywhere to use NTLM with SSL required by default.
  • If you have different auth requirements for Outlook Anywhere use the optional parameters to set those.
  • The script sets PowerShell to use Basic with SSL required by default.
  • If you have different authentication requirements for PowerShell use the optional parameters to set those.
  • PowerShell was added to the settings. Please be sure to REM those lines of code should you NOT want to configure the PowerShell Virtual Directory.

USAGE:

HAPPY SCRIPTING!
POSITIVE ENERGY!
PLEASE COMMENT!

REFERENCES:
Exchange Server Client Access URL Configuration Script
PowerShell Script to Configure Exchange Server Client Access URLs

Author Lance LingerfeltPosted on

Installing an ‘IP-less’ Exchange Server 2019 Database Availability Group

Yesterday, I posted on how Exchange now uses the Resilient File System (ReFS) to optimize and protect Exchange critical files. Another layer of protection is using a database availability group (DAG) for redundancy and is a necessary factor when designing an Exchange Enterprise Environment.
In this example, I will walk you through the installation of an Exchange Server 2019 DAG as I configured in my environment. This DAG will contain two Exchange Servers in the same site with a third Windows Server 2019 server being the File Share Witness (FSW).

Two Server Exchange DAG Configuration

For my configuration, I configured two identical Windows Server 2019 VMs (same procs, RAM, vhdx drives, partitions, etc…). I configured the Exchange Data Volume using ReFS and mounted them to the same folder on the C: Drive on each server. This is very important for replication to take place successfully when the databases are added to the DAG.


I next went to the Admin server where the FSW would be hosted and added the Exchange Trusted Subsystem Account to the local Administrators group on that server:

IMPORTANT!
Add the Exchange Trusted Subsystem Account to the Local Administrators Group on the FSW.

NOTE: The reason that this is an ‘IP-less’ DAG is that I’m creating a DAG with no cluster administrative access point (CAAP). The DAG has no IP address of its own, and no computer object in Active Directory. The main implication of this is that backup software that relies on the CAAP or backup operations won’t work. This option of an ‘IP-less’ DAG was first introduced in Exchange Server 2013 SP1/CU4, so by now any decent backup products should support this configuration. But you should always verify this with your backup vendor of choice. Also be aware that this is only supported for DAGs that are running on Windows Server 2012 R2 (or later).

Next, we create the DAG from Exchange PowerShell using the New-DatabaseAvailabilityGroup cmdlet. Now remember that since you are using the ReFS system for your database volumes, you will need to specify the -FileSystem parameter within the cmdlet to assure proper setup and replication of the data files.

Next, we add the Exchange Servers that hold the databases that will be replicated within the DAG:

The DAG will now show the two servers as Operational Member Servers:

The FSW Directory was created on the admin01 server when the DAG was created. We can verify that with the following cmdlet:

Next, we add the databases that we want replicated to the DAG as replicated databases. I want all my Databases on EX01 to replicate to EX02 and vice versa for the EX02 Databases. I want the activation preference to remain on the server that the databases were originally created on so I will use the -ActivationPreference parameter to accomplish that. I will go into more detail on Activation Preference in another post.

Now we verify that the Database Copies are healthy on each replication member using the Get-MailboxDatabaseCopyStatus cmdlet. You will see a Healthy Status on the replicated copies:

POSITIVE ENERGY!
KILL NARCISSISM!
HAPPY TROUBLESHOOTING!

REFERENCES:
Installing an Exchange Server 2016 Database Availability Group

Author Lance LingerfeltPosted on

Using the Resilient File System for Exchange Server

In my ongoing effort for becoming more knowledgeable on Exchange Server, I found that the preferred new file system for Exchange Databases and Log files is the ReFS.
ReFS is not that new. Microsoft’s Resilient File System (ReFS) was introduced with Windows Server 2012. ReFS is not a direct replacement for NTFS, and is missing some underlying NTFS features, but is designed to be (as the name suggests) a more resilient file system for extremely large amounts of data.

Support for ReFS with Exchange Server

From Exchange Server 2013 and upwards (which includes Exchange Server 2019 today) Microsoft supports the use of ReFS for Exchange servers, and in fact they now recommend it as the preferred file system for Exchange Server 2019, within the following guidelines.

For Exchange Server 2013:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features must be disabled for the database (.edb) files or the entire volume that hosts database files.
  • Hotfix KB2853418 must be installed.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2013 installation volume, but you can consider using ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2016:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2016 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2019:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.

This means that you should continue to use NTFS for your operating system and Exchange Server 2019 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

Creating an ReFS Formatted Volume

In Windows Server during the New Volume Wizard when you get to the step for configuring File System Settings change the file system from NTFS to ReFS.

exchange-server-refs

NOTE: Using the New Volume Wizard does not give you the option to disable data integrity at the volume level. To set it at the volume level itself use PowerShell when configuring new volumes. I found this out the hard way and am now re-configuring my volumes to disable the Integrity Streams.

I needed to create the mount point to mount the volume to:

I then got a list of my available disks:

In my case, disk 2 was the one I needed to format and change. I had to create a new partition and then format it:

Once formatted, I mount the volume to the Directory created earlier:

NOTE: Partition 1 on a disk is always reserved for system files on the drive volume. So the active partitions will always start at 2.

Lastly, verify that the partition is online and that the Integrity Streams are turned off:

Additional Considerations

When you are deploying an Exchange 2016 or 2019 DAG and using Autoreseed, the disk reclaimer needs to know which file system to use when formatting spare disks. So when, creating a DAG in Exchange PowerShell, make sure to set the -FileSystem parameter. For Exchange Server 2013 DAGs, manually format the spare volumes with ReFS.

More coming soon. I will post how I setup the “IP-less” DAG for my environment and got replication functional for my Exchange Databases.

REFERENCES:
Exchange 2013 storage configuration options
Exchange 2016 Preferred Architecture
Exchange Storage for Insiders: It’s ESE (Ignite video)
ReFS Exchange Server Volumes
Preparing ReFS Volumes for Exchange

Author Lance LingerfeltPosted on

Importing User Photos to Office 365 in bulk for your company.

In a previous post, I showed how you could update one user’s photo for their Outlook and AD profiles via PowerShell. In this post, we will explore how to do this for your entire organization via PowerShell to Office365.

NOTE: I have not tested the scripts as I do not have enough mailboxes in my O365 tenant along with not using a ‘.’ in my alias. If the scripts are incorrect, please inform me with the correction and I will update accordingly.

Please make sure that your photos are reviewed before posting, and try to keep the file size of the photos to a minimum. In Office 365, there exists a limitation for the user photo not to be more than 10 KB in size, but I will show you how to get around that limitation.

Having a user photo for each of your users is very beneficial as it personalizes each account to a face in the company. The user photos can be viewed in below locations:

  • Outlook Web Access
  • Contact Card
  • Thumbnail in emails
  • Outlook Client
  • Yammer
  • Lync Client
  • SharePoint (People Search / Newsfeed)

Steps to take:

  1. Remove the 10KB photo size limitation in Exchange Online
  2. Prepare a folder with all users photos
  3. Update the profile photos via a PowerShell cmdlet.

Connect to Exchange Online with the RPS Proxy Method to remove the 10K size limitation

NOTE: In the PowerShell cmdlet above, we connected using a different proxy method. This was to overwrite the limitation of uploading the images with size more than 10KB. Using the different proxy method (/?proxyMethod=RPS ) to connect to Office 365 in the above cmdlet accomplishes this.

Prepare a folder locally and place all the photos in that folder

Create a folder named C:/UserPics and make the filename of each photo be the username of that particular user. (i.e. llingerfelt.png)
The below script should be able account for aliases that have a ‘.’ in the id as well. (i.e. lance.lingerfelt)

NOTE: From my research, there is no set photo type that is required for the photo. My suggestion would be to keep the photos .png for size constraints while maintaining picture clarity.

Update the profile pictures via PowerShell

Create the following script and name it Photos-Update.ps1

Run Photos-Update.ps1 and the script should upload the photos to Office 365 and apply each photo to the corresponding user.

NOTE: If you’re still having some issues with the alias having a ‘.’ in the name, you can also configure the Photos-Update.ps1 script in this manner to get that working properly:

HAPPY SCRIPTING!
PLEASE COMMENT!

store.ldlnet.net
LDLNET LLC! Your Source for Pofessional IT Services!
www.servermonkey.com
Contact ServerMonkey.com for your IT Hardware Needs!

REFERENCES:
How to import Office365 User photos over 10KB & without CSV in bulk

Author Lance LingerfeltPosted on

Set the profile pic for a single Exchange user via PowerShell

I wanted to update my picture within my Outlook profile and AD account really quickly without having to go through OWA to do so. I found this cmdlet that will allow for that picture to be changed very quickly via Exchange PowerShell.

NOTE: This can be done with On-Premises Exchange and Exchange Online PowerShell

Old picture within my account

First, download the picture you want to use to the computer that you want to run the cmdlet from. Also, make sure the picture is cropped and centered prior to running the cmdlet. I saved the pic to C:\temp for my scenario. The best format to use would be jpg. I named the file User1_Profile.jpg

Next, open Exchange PowerShell on the computer you saved the pic to and run the following cmdlet to change the photo:

Once completed, the Outlook client should be closed and reopen so that the new picture is visible in the profile.

Picture change completed

I will post how to perform this for multiple users for Exchange and Office365 in a later post.

REFERENCES:
Set User Photo with Exchange PowerShell

Author Lance LingerfeltPosted on

Purging Soft Deleted mailboxes from Exchange Server

If you’re a seasoned administrator, you have knowledge that in Exchange, the database settings will allow you to set the deleted mailbox retention. The default is 30 days, but sometimes you need to purge all those deleted mailboxes to do some ‘spring cleaning’ as it were. Note that doing these cmdlets does not change the ‘Whitespace’ of the database or the size. In my case, I had to purge everything of a toxic individual that was tainting my network much to my disappointment and did the following to complete that task.

The following cmdlet will seek all Soft Deleted mailboxes within the database you select and manually purge them from Exchange.

Now, should you only want to remove one mailbox, you will need to get the GUID of that Soft Deleted mailbox first so that you can enter it for the identity parameter.

You can also preform a similar task for a disabled mailbox:

You can perform the task on all disabled mailboxes for that database as well:

NOTE: I would be very careful when performing either of these cmdlets as they will completely purge the mailboxes from the schema. If these cmdlets assist you with your ‘spring cleaning’, I will have been happy to assist.

HAPPY PURGING!
PLEASE COMMENT!
IGNORANCE IS NOT BLISS!

References:
Purging Deleted Mailboxes on Exchange 2013

Server Monkey
LDLNET LLC’s Preferred Server Equipment Hardware Vendor
Author Lance LingerfeltPosted on

Customize your Outlook Web App Logon Page

As many of you are aware, Microsoft provides a default logon page for OWA, the Outlook Web App. Most companies, like myself want to be able to customize that page so that it suites your organization. Here is what my company OWA page looks like:

Customized OWA Logon Page

I have changed the color on the left to match my scheme, replaced the Outlook Logo with my company logo, and added a disclaimer to notify users. Below is the process to do that effectively for your organization.

NOTE: Every time you install an Exchange Cumulative Update (CU) or new version of Exchange Server these modified files will be replaced. Remember to backup your original and changed files to another folder so that you can replace them when you Update or Upgrade or if something goes wrong with the changes.

Customize the color of the Outlook on the web sign-in page

  • Use Notepad to open the file:

%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources\logon.css

  • In the logon.css file, replace the default blue hexidecimal color value #0072c6 with the HTML RGB value that you want to use. You can use the following LINK to choose the color you wish to use.
  • When you’re finished, save and close the file.

Here are the different graphics that can be changed on the OWA logon page and their associated files:

Outlook on the Web sign-in page with element call-outs
ImageFile nameLocationDimensions (width x height in pixels)Bit depth
1 
favicon.ico 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
16 x 16 
32 
2 
olk_logo_white.png 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
128 x 108 
32 
3 
owa_text_blue.png 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
300 x 76 
32 
4 
Sign_in_arrow.png (for left-to-right languages) 
Sign_in_arrow_rtl.png (for right-to-left languages) 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
22 x 22 
32 
  • Just resize your images to the given dimensions in the table, rename them to the file name, and replace the files in the directory.

Change the disclaimer text for your OWA logon page

Next, we want to add a disclaimer to our logon page. To do that, we need to modify the logon.aspx document in the following directory:

%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\logon.aspx

Open the file in Notepad or your favorite HTML editor and search for the text ‘hidden-submit’. When you find the text, you can add your disclaimer text under the div class=”disclaimer” tag as I did in the following example:

Save your logon.aspx file and give your OWA server an IISRESET for good measure. You should be good to logon with the new page from that point on.

HAPPY CONFIGURING!
PLEASE COMMENT!
THANKS FOR YOUR SUPPORT!

References:
Customize the Outlook on the web sign-in, language selection, and error pages in Exchange Server
CUSTOMIZE EXCHANGE 2016 OUTLOOK ON THE WEB SIGN IN PAGE
Customizing Exchange 2016 OWA

Author Lance LingerfeltPosted on

Installation of Exchange Server 2019 on Windows Server 2019

I have realized recently that I am an Exchange Messaging Professional, but yet, I have not posted the methodology of how I install an Exchange Server Mailbox Role. So here it is!

Install Windows Server 2019

Exchange Server 2019 requires Windows Server 2019 to run. For my environment, I haven’t necessarily need to follow all the enterprise level design aspects of database numbers to mailbox size ratios, number of servers, front/back end configurations, DAG Implementation, etc… If you want or need to delve into that realm, you can go here. I have need for a single server with only a few databases for a small number of mailboxes, so I am approaching it from that standpoint.

So first, in Hyper-V, I configured my VM with the following specifications:

Processors: 2 procs with 2 cores each – 4 Virtual Processors Total
RAM: 32GB with dynamic memory enabled optional
Drives: 2 .vhdx drives of 120GB each (OS / Exchange Data)
CD: Windows 2019 ISO
Default Settings for the rest of the VM Settings

Next I installed Windows Server 2019 Datacenter with the GUI! You can install it on Server Core if you wish. That information can be found in this link.

I ran through the setup of Windows and installed the OS on my first vhdx drive. I booted up, set the local admin password, and logged in. Once in Windows, I went to the Local Server Settings in Server Manager and configured the following settings:

Set the Date, Time, and Time Zone. (Once in the Domain, this would sync through Group Policy)
Set IE ESC to allow Administrators to have full IE access.
Set Remote Desktop Settings to gain RDP access. (This would be locked down with Group Policy as well once on the Domain)
Set the IP Settings to Static Settings. (DNS Servers, Gateway, WINS, etc…)
Join the server to the Active Directory Domain.
Reboot the VM Server.
Logon to your Domain.
Configure Windows Update Settings. (I have WSUS through Group Policy, this was configured automatically upon reboot)
Download and install all Windows Updates for the server. Then Reboot.
Open Disk Management and configure the secondary vhdx drive to be your Exchange Data Drive.
I configured the drive to be a mounted folder ‘C:\Exchange\Data’ rather than another drive letter as that seems to be the more accepted form of installation for the data drive these days. That is based on the multiple configurations that I have seen for Exchange through experience in Enterprise environments. Again, to each is own and depending on you design specifications, you might want to do that differently.

Next, we need to install the prerequisites for the Exchange Mailbox Server. I have always used practical365.com to get the PowerShell script to install the prerequisites, but couldn’t find the article this time. Great site though! Instead, I got the information and ran the following from an elevated PowerShell Session locally on the server:

As part of the prerequisites you will need to install the following packages onto the server as well:

UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013

Once completed, you can begin the install of Exchange. If this is your first Exchange 2019 Server in your Organization, then you will need to run the following to update the Forest, Schema, and Domain so that Exchange will install properly:

NOTE: If you run into Prerequisite issues with the installation due to a “pending reboot”, check out my blog post for information on remediation of that issue.

Now that the environment is prepared for Exchange, you can actually begin the installation. I wanted to make my default database and logs folder to be on the Exchange Data volume that I created, so I included those settings in the setup command. Please look at the reference to the setup.exe switches for more information on that. Here is the command:

Setup should go through the installation via the PowerShell window and complete successfully. Reboot the Exchange Server, then you can then logon to the Exchange Admin Center and begin the process of configuration of how you need to integrate the Mailbox Server into your Server Farm. That configuration is for a later post.

PLEASE CHECK BACK FOR UPDATES!
PLEASE COMMENT!

References:
UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013
Install Exchange Server 2019 on Windows Server 2019 Core
Exchange Server Design Planning
Use unattended mode in Exchange Setup
Practical365 on Exchange 2019

Author Lance LingerfeltPosted on

Exchange Hybrid Configuration Wizard Link

Wanted to do a quick post as I was working on my Hybrid Exchange Environment. I was unable to get the HCW to download and start from the Exchange Control Panel with the link provided on the page. This has happened to me for a while, so I went online and found a link that would work that could be downloaded and reused to open the HCW:

Hybrid Configuration Wizard Link

HOPE THIS HELPS!
LET ME HAVE KNOWLEDGE SHOULD THE LINK CHANGE!

References:
HYBRID CONFIGURATION WIZARD WON’T START ON WINDOWS 2016

Author Lance LingerfeltPosted on

Exchange Setup Repeatedly Says ‘A Restart from a Previous Installation is Pending’

I have had this issue with EVERY upgrade that I have ever attempted for Exchange Server from 2013 through 2019 CU1. You go to run the setup program and during the prerequisite checks, setup stops. The error listed is:

A restart from a previous installation is pending. Please restart the system and rerun setup.

During the prerequisite checks, Exchange Setup looks in the registry at the following keys:

  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Updates\UpdateExeVolatile
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations

Nine times out of ten, a restart does NOT remediate this error. In order for setup to continue properly, you must do the following:

  • Open regedit: Start > Run > regedit.exe
  • Set the HKLM\SOFTWARE\Microsoft\Updates\UpdateExeVolatile key value to 0 or delete it if present. <– This one is usually NOT present.
  • Delete the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations key.
  • Re-run Setup.

You should now be able to run setup and upgrade your Exchange Server.

PLEASE COMMENT!
HAPPY TROUBLESHOOTING!

References:
A Restart From Another Installation Is Pending
Exchange Setup Fails – A Restart From Another Installation Is Pending
Microsoft Document – A Restart From Another Installation Is Pending

Author Lance LingerfeltPosted on

Reconnecting Shared Mailboxes after an O365 Migration

I get a lot of these incidents in my queue after a user has been migrated to O365. For whatever reason, most likely due to the mailbox being moved itself, whether it is the user’s mailbox, the shared mailbox, or both, the connections to the shared mailboxes stop working in Outlook and the user cannot connect to the shared mailbox.

Here is a quick and easy solution to use to disconnect and reconnect the shared mailbox(es) that you lose connectivity to when migrated. This is usually performed on Outlook 2016 and above as most users upgrade their client software when moved to O365.

First, we remove the existing shared mailbox connection:

  • Click the File > Account Settings > Account Settings.
  • Select your company email address in the account list.
  • Click Change > More Settings > Advanced tab > Select the Shared Mailbox > Remove
  • Click Apply > OK > Next > Finish.
  • The shared mailbox will now automatically be removed in your Folder pane in Outlook.

Second, we re-add the shared mailbox connection to Outlook:

  • Click the File > Account Settings > Account Settings.
  • Select your company email address in the account list.
  • Click Change > More Settings > Advanced tab > Add
  • Type the name of the shared mailbox in the window and click OK.
  • Click Apply > OK > Next > Finish.
  • The shared mailbox will now automatically be added to your Folder List pane within Outlook.

Note: The above procedure must be followed in order to properly reconnect the shared mailbox. You cannot remove and re-add the mailbox in the same process as that will not reset the connection properly. You must save the settings when disconnecting.

I hope that this will assist everyone when troubleshooting Outlook connectivity issues to shared mailboxes after a migration.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!

Author Lance LingerfeltPosted on

Moving mailboxes to O365 via PowerShell in Hybrid Configuration

As many of you have knowledge, I am studying for my MS-202 Exam. And, part of the knowledge needed is to be able to migrate mailboxes between on premises and Exchange Online through PowerShell. Here are the steps for the scenario to move a mailbox from on premises to O365:

1. Connect to Exchange Online via PowerShell

If you have read my previous post: Connect to All PowerShell Modules in O365 with one script
You should have all the settings needed to connect your PowerShell to O365. Note in this scenario, that all these cmdlets will be run from O365 PowerShell and will be monitored from O365 by either PowerShell or the Exchange Admin Center. You will not be able to monitor the moves from On-Premises.

2. Provide your on premises Migration Administrator credentials as a variable for your cmdlet.

3. Move a single mailbox.

In your hybrid configuration you should be doing directory sync with O365/Azure and the accounts should be available in the cloud showing that they are synced with AD. This also assumes that you have your MRS Proxy endpoint enabled, which can be done by the HCW. Also, make sure you have your licensing available for your mailboxes. From my knowledge, you can assign your license to the account in the cloud before moving, especially if you have a particular license that you need to assign the account. Other than that, moving the mailbox will assign an existing license that is available that includes an Exchange Online mailbox feature when the mailbox is moved.
Now we initiate the move with the cmdlet. Similar to what you would do in the GUI, this simple mailbox move cmdlet initiates the move request. It has most of the same parameters as a local move request including BadItemLimit, LargeItemLimit, AcceptLargeDataLoss, etc…
 
Use the following LINK for documentation on the New-MoveRequest cmdlet.

Now with all migration projects, we expect to have to move multiple mailboxes in a single batch. The following will show the process for moving mailboxes in bulk from on premises to O365:

1. Connect to Exchange Online via PowerShell

If you have read my previous post: Connect to All PowerShell Modules in O365 with one script
You should have all the settings needed to connect your PowerShell to O365. Note in this scenario, that all these cmdlets will be run from O365 PowerShell and will be monitored from O365 by either PowerShell or the Exchange Admin Center. You will not be able to monitor the moves from On-Premises.

2. Provide your on premises Migration Administrator credentials as a variable for your cmdlet.

3. Move multiple mailboxes in a single batch.

In your hybrid configuration you should be doing directory sync with O365/Azure and the accounts should be available in the cloud showing that they are synced with AD. This also assumes that you have your MRS Proxy endpoint enabled, which can be done by the HCW. Also, make sure you have your licensing available for your mailboxes. From my knowledge, you can assign your license to the account in the cloud before moving, especially if you have a particular license that you need to assign the account. Other than that, moving the mailbox will assign an existing license that is available that includes an Exchange Online mailbox feature when the mailbox is moved.

This time you want to create a CSV file using the alias or emailaddress as your header and then list the appropriate value for all the users in your batch group. Save the file locally as MigrationBatch01.csv or a name of your choice.

Use EMailAddress
 OR
 Alias as the header

Next you initiate the mailbox moves. When specifying the mailbox identity in the cmdlet, use the respective header in your variable declaration (either $user.EMailAddress OR $user.Alias)

Use the following LINK for documentation on the New-MoveRequest cmdlet.

GOOD LUCK WITH YOUR MIGRATIONS!
HAPPY TROUBLESHOOTING!

References:
Moving Individual Mailboxes to O365
Move Mailboxes in Bulk to O365
PowerShell Mailbox Migration to O365
Connect to all PowerShell Modules in O365 with one script
New-MoveRequest Microsoft Document

Author Lance LingerfeltPosted on

Exchange DAG Replication Problem: An established connection was aborted by the software in your host machine

I had an issue with a four node DAG where the DR site with two of the DAG members were having replication issues. It was only technically affecting one DAG Member though. The copy queue length was really high and the logs were not committing to the database. A Test-ReplicationHealth cmdlet test told that the copy queue length for the affected database copy was high. No other databases were affected as there were eight databases on this DAG Node. The issue was that the log files were not replicating properly to the one DAG member for that database, causing the log file drives on all the other DAG members to build and become full:

EX04 DAG member has high Copy Queue Length
The purple member (EX04) free space is different from the other three DAG members

Circular Logging was turned on, but since the db was NOT in sync, the logs could NOT truncate properly which rendered CL useless. What was being done to stave the issue was to suspend the database copy of the affected DAG member (EX04), then resume the copy. The logs would replay and commit to the database copy on the DAG member, but over a short period of time, the same issue would arise again, as shown in this graph:

You can see the other DAG members start dropping in free space

There were absolutely no errors in the Event Viewer showing this replication issue. After some research, I ran the following cmdlet showing a particular output parameter that gave me the actual problem:

Get-MailboxDatabaseCopyStatus DAG1DB01 | ft -a -wr Name, Status, IncomingLogCopyingNetwork

Output with the actual error listed for the DR DAG members.

The operative error here was: {An error occurred while communicating with server ‘EX01’. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.} 

Now even though only EX04 was actually having problems with its log replication, both DR members EX03 & EX04 were having the same problem. Again, there were NO events in event viewer showing this issue. I next did some connectivity tests to EX01 from EX04 even though the error said there was an established connection that was broken.

Ping EX01 -f -l 1472

Now the -f states do NOT fragment the packet and send it as a whole to the destination.
The -l states the packet/buffer size you want sent. In this case 1472 bits.
By doing this, you are able to assure that a router or switch is NOT segmenting the packets, packet segmentation of replication logs can cause data corruption and replication issues.

That test passed successfully. I also did a trace route to assure there was no packet loss on the route to the replicating server. That test passed successfully.

I next checked the DAG Network to assure that all networks were working for replication. Now, in this scenario, there was only ONE DAG Network, there was NOT a separate Replication Network. I did not design the DAG and limitations most likely came into play during the design. From my experience, you setup a separate replication network for replication only, but if your network has enough bandwidth, and the design calls for simplification, you can use one DAG network in your design.

Get-DatabaseAvailabilityGroupNetwork | fl 

RunspaceId : a1600003-8074-4000-9150-c7800000207f 
Name : MapiDagNetwork 
Description : 
Subnets : {{192.168.1.0/24,Up}, {192.168.2.0/24,Up}} 
Interfaces : {{EX01,Up,192.168.1.25}, {EX02,Up,192.168.1.26},{EX03,Up,192.168.2.25}, {EX04,Up,192.168.2.26}} 
MapiAccessEnabled : True 
ReplicationEnabled : True 
IgnoreNetwork : False 
Identity : DAG1\MapiDagNetwork 
IsValid : True 
ObjectState : New 

All the DAG Network Members were up and not showing errors. I next did a telnet session to EX01 over the default DAG replication port 64327 to see if there would be any connectivity issues to EX01:

telnet EX01 64327

That test was successful and there were no connectivity issues to EX01 from EX04. Again, there was only ONE database out of eight that was having replication problems. After mulling over the problem, it was decided to restart the MSExchangeRepl service on EX03 AND EX04 since the error was present on both DAG members. We would then, suspend the database copy and resume the database copy on the affected servers.

Run on EX03:
Restart-Service MSExchangeRepl
Suspend-MailboxDatabaseCopy DAG1DB01/EX03 -Confirm:$False
Resume-MailboxDatabaseCopy DAG1DB01/EX03 -Confirm:$False

Run on EX04:
Restart-Service MSExchangeRepl
Suspend-MailboxDatabaseCopy DAG1DB01/EX04 -Confirm:$False
Resume-MailboxDatabaseCopy DAG1DB01/EX04 -Confirm:$False

After monitoring the databases and log drives, the issue was resolved and replication started functioning properly.

Log Drive Available Space Returned to Normal for DAG members

PLEASE COMMENT! I WELCOME SUGGESTIONS, TIPS, ALTERNATIVE TROUBLESHOOTING! HAVE A GREAT DAY!

Author Lance LingerfeltPosted on

Connect to all PowerShell Modules in O365 with one script

Let’s say you’re an admin that needs to connect to Office365 via PowerShell often. Now, there are many different websites or blogs that will show you how to connect to each session via PowerShell. That can cause a headache since you can end up having five different PowerShell sessions running in five different windows. You end up having to enter a username and password all those times, which can become time consuming.

I want to show you here how to combine all those sessions into one script where, if you’re security is tight enough on your computer, you don’t even have to enter credentials. This way, you can click on one icon and pull up all the O365 PowerShell commands that you’ll need to manage your organization.

First you need to download the following PowerShell Module Installation Files so that your PowerShell Database will have the correct modules installed:

Microsoft Online Service Sign-in Assistant for IT Professionals RTW
Windows Azure Active Directory Module for Windows PowerShell v2
SharePoint Online Management Shell
Skype for Business Online, Windows PowerShell Module

Next, we want to setup the CLI (Command Line Interface) to be too cool for school. I have learned it helps to have knowledge of how to customize the CLI window. You can do all of this in PowerShell ISE or Notepad, which ever you prefer. Here are the commands for the script that I use to setup the CLI:

Next, you want to set your Execution Policy and put in your credentials so that you won’t be prompted to enter the user credentials when you run the script.

NOTE: MAKE SURE YOU KEEP YOUR SCRIPT SAFE AS THE CREDENTIALS ARE VISIBLE WITHIN THE SCRIPT IN PLAIN TEXT!

You can, alternatively, set your script to prompt for credentials every time by using the following:

$LiveCred = Get-Credential

Here is that part of the script:

Now we get into the importing of the modules for each O365 service:

Get the MSOnline Module:

Connect to the MSOnline Service:

Connect to Azure AD PowerShell:

Connect to SharePoint Online PowerShell:
NOTE – MAKE SURE YOU CHANGE TO YOUR COMPANY NAME IN THE URL!!

Connect to Exchange Online PowerShell:

Connect to Skype For Business Online PowerShell:

Connect to the Security & Compliance PowerShell:
NOTE – This one I still get “Access Denied” when trying to connect. I have looked for an answer to that issue, but have not found one. Please comment with a link if you have an answer so that I can update this script!

Lastly, put in a note to show that the PS load is completed:

So Here is the final script in its entirety:

Now you can create your icon for your desktop so that you can easily access the script. I would save the script to your Scripts directory.

That will usually be C:\Users\’username’\Documents\WindowsPowerShell\Scripts or wherever directory you choose.

To start, right click the desktop and choose New > Shortcut
In the Target Field, enter the following for your PowerShell Shortcut, pointing to the path of your script:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -ExecutionPolicy Unrestricted -File “C:\Users\username\Documents\WindowsPowerShell\Scripts\ConnectO365All.ps1”

Click on the Advanced button and check the box: Run As Administrator
Under the General Tab, name your shortcut: (CompanyName) O365 All PowerShell
Click OK to save the shortcut to your desktop.

LAST BUT NOT LEAST, RUN THE FOLLOWING COMMAND BEFORE EXITING OR CLOSING YOUR POWERSHELL WINDOW. THIS WILL REMOVE ALL THE SESSIONS YOU’VE CONNECTED TO:

Get-PSSession | Remove-PSSession

HAPPY SCRIPTING!
LEARN, DO, LIVE!

References:
Connect to all O365 Services in one PowerShell Window
How to connect to all O365 Services through PowerShell
Connecting to Office 365 “Everything” via PowerShell

Author Lance LingerfeltPosted on

Checking Drive Space Volumes for DAG DB members through PowerShell

I had received a weird alert for a DB volume for a DAG member being below threshold. This was odd to me due to the fact that there were four DAG members and we only received an alert for one. I went into Azure Log Analytics and ran the following query to render a graph for the past 14 days showing the percent free space of the volume for all the DAG members.

Thanks Georges Moua for the query script!

Now the reason I can run the query this way is due to the fact that the Design of the DAG was correctly done and the DB folders are identical on all DAG members. The query rendered the following chart:

As you can see the Green DAG member is way below the other DAG members.

I next went to an Exchange Server in the DAG and got the volume data for all the members in the DAG:

EX02’s volume free space is far below the other DAG members

I went on EX02 and found that there was a subfolder named “Restore” that was not present on the other servers. I ran the following script to get the size of that folder in GB:

The folder size was 185 GB. Removing that folder, along with all subfolders/files, would balance the free space to the other DAG members. I ran the following cmdlet to remove the folder and all subfolders/files:

This remediated the alert and balanced the drive space across all DAG members.

POST YOUR COMMENTS OR QUESTIONS!
HAPPY TROUBLESHOOTING!

Author Lance LingerfeltPosted on

Event 11022 with MSExchangeTransport – Easy Validation Test

In a hybrid environment, you’re always connecting between the cloud and on premises to establish transport through the connectors to transport mail. By default, this is done over a TLS (Transport Layer Security) connection. It’s similar to a VPN or SSL connection using certificates on the Transport Layer of the network stack to encrypt the data between the two Organizations in a Hybrid configuration.

Because you are using certificates, the certificate must be validated properly and checked to see if it has expired or been revoked by the issuing company. A revocation list is created and updated regularly for this purpose. If the connecting organization cannot validate the revocation of the certificate, it will not establish a TLS connection with the connecting organization. You will then get the following event:

Event 11022
MSExchangeTransport
Error:
Failed to confirm domain capabilities ‘mail.protection.outlook.com:AcceptOorgProtocol’ on connector ‘Inbound from Office 365’ because validation of the Transport Layer Security (TLS) certificate failed with status ‘RevocationOffline’. Contact the administrator of ‘mail.protection.outlook.com’ to resolve the problem, or remove the domain from the TlsDomainCapabilities list of the Receive connector.

Most likely, there is a network issue with the On Premises Organization being able to retrieve the Revocation File with the Certificate Information. Since it cannot retrieve that file, it stops the transport connection and throws the error.

A simple validation to validate the connector and assure transport from Office365 is to run the following cmdlet from the server on premises that performs the connection:

Again, I like to put the other cmdlets of 
write-host, hostname, and date 
in order to make it easy to document when working an incident.

From the highlighted text, we can see the test was successful.

The test runs a connection for each connector and tests the validity of each connector. If a success is returned, then we have knowledge that the certificate was validated and the connection was established through the connector from Office365.

If you get a failure though, you will need to run tests to see if you can pull the revocation list for the certificate as well as a simple test to connect to Office365:

Connect to Exchange Online via Powershell

IMPORTANT NOTE

I wanted to put some information on how to pull the CRL Distribution Point for the Office365 so that you could run an Invoke-WebRequest to pull the CRL file from the Distribution Point, but I have NOT found a single way through Powershell to pull that information. I have searched multiple posts and articles showing all these advanced methods of using certutil and PowerShell to get a bunch of other information, but NOTHING on how to pull the URL for the CRL file from the certificate. Doing a Get-ChildItem for the certificate using the Thumbprint does NOT pull that property from the certificate. Now, if you have a cmdlet that WILL do that, PLEASE POST!

So, in essence, to troubleshoot if you can get to the CRL file, you get the URL for the CRL Distribution Point from the GUI Properties of the certificate. Then you run the following cmdlet in PowerShell:

POST COMMENTS!
HAPPY TROUBLESHOOTING!

Author Lance LingerfeltPosted on

What the Hybrid Configuration Wizard Performs in the background and configuring Hybrid Co-Existence with Exchange Online

I’m working on getting certified in Exchange Hybrid Scenarios and Exchange Online configuration as part of my skill set for Exchange. In doing so, I had successfully implemented a complete Full Hybrid Exchange Environment between my Exchange Online Tenant and my On Premises Exchange 2019 Environment last evening.

I wanted to give an update that was posted to my LinkedIn Posting on this. Thank you Brian Day for the vote of confidence and caution that running these cmdlets manually is not supported by Microsoft and that the HCW, like all the Online Microsoft Products, is constantly changing and being updated.

Important Note

As preparation, I bought some Exchange Online Plan 1 licenses which give me a 50 GB mailbox limit and basic mailbox functionality. It does not include the more advanced features such as ATP, or DLP. I am running most of those features through my On Premises Environment. I mainly wanted to be able to place mailboxes in the cloud and have a hybrid setup. My plan was to have mail flow continue through my On Premises environment so that my Exchange Server features would be used and I would not have to change any MX or SPF records. I also had my certificates in place for SSL and OWA so I would want keep mail flow routed that way, through on premises. I do want to be able to have Free/Busy lookups cross-premise so federation would have to be enabled as well. I would also have to enable the MRS proxy on my Exchange Server so that mailbox migration could be implemented cross-premise. I also have previously configured Azure AD Sync along with ADFS for Single Sign On. In my case, another server was not needed as I didn’t have enough mailboxes or real need to split my frontend and backend deployment. Running the Hybrid Configuration Wizard would not open any new ports or change any existing port traffic that was already configured on my firewall. These are just a few of the considerations that need to be looked at when considering a hybrid integration.

Here is a great article to read for the prerequisites
Exchange Hybrid Deployment Pre-requisites

So, once I had all those considerations handled in my design, I ran the Hybrid Configuration Wizard. What I want to do in this blog post is to go through the steps that the wizard does in the background to setup the Hybrid Environment as you go through the Wizard.

I mainly used the following blog post as a reference, but have approached it differently by diving into the cmdlets that are run during the process:

Exchange/Office 365 Hybrid Configuration Wizard – step by step guide

1. The HCW validates the On-premises and Online Exchange Connection.

The Hybrid Configuration Wizard checks if it is possible to connect to both servers with PowerShell. It runs the Get-ExchangeServer cmdlet on premises after resolving the server in DNS. It then connects to Exchange Online, authorizing the connection:

Authority=https://login.windows.net/common Resource=https://outlook.office365.com ClientId=abcdefgh-a123-4566-9abc-2bdflancelin

2. The HCW collects data about Exchange configuration from the on-premises Active Directory

The Wizard gathers information about the local domain. In order to do that, the HCW executes a series of cmdlets.

These include, in order:

3. The HCW collects information on the Exchange online (Office 365) configuration

This task repeats what has been done in the previous step, only for the Exchange online, instead of the on-premises one.

The cmdlets include, in order:

4. Federation Trust is determined. If not present, a new Federation Trust and the required certificate will be created on the local Exchange Server

You will be prompted in the Wizard to create a Federation Trust if not present. The following articles explain Federation and its requirements:

Understanding Federation – Link Here
Understanding Federated Delegation – Link Here
Create a Federation Trust – Link Here

If the activity is finished successfully, a new certificate should appear on the on-premises Exchange Certificates list. The new certificate includes “Federation” in its Subject field. To make sure the certificate is there, you can run a cmdlet: Get-ExchangeCertificate | ft -a -wr


The results will look like this

5. The HCW creates a new Hybrid Configuration Object in the local Active Directory

The HCW will run cmdlets based on the information you provide in the HCW for the certificate, the on premises Exchange Server, the domain(s), and what features you want turned on:

It then checks the settings through the following cmdlets:

It then enables Organization Customization for both environments through this cmdlet:

6. Configuration is then completed to modify the settings on the on premises Exchange environment 

EmailAddressPolicy – HCW adds address @tenant.mail.onmicrosoft.com
The HCW configures remote domains – adds tenant.mail.onmicrosoft.com and tenant.onmicrosoft.com
The HCW adds a new accepted domain – adds tenant.mail.onmicrosoft.com

Some of the cmdlets run:

7. The HCW Configures the Organization Relationship between the local server and the cloud.

This configuration is not necessary in minimal hybrid deployment. Since I have a full hybrid deployment configured, the cmdlets were run as needed to configure it. Thanks to the correct configuration, it is possible to synchronize free/busy status of mailboxes and their elements between the on-premises Exchange Environment and Exchange online. 

Some of the cmdlets run in the process:

8. The HCW and setting connectors on both Exchange servers
The HCW checks to see if the connectors are there, if not, it sets them up. During this workflow, four connectors are set – one receive and one send connector for each server. Those connectors guarantee the mail flow between the on-premises and Exchange Online.

Some of the cmdlets run in the process:

The Intra-Organization is set as well:

9. The HCW configures OAuth Authentication across the Hybrid

This LINK explains how OAuth is configured between Exchange On Premises and Exchange Online. It’s a very good article to read as it shows how to get the Modern Authentication style working. Now the HCW does this for you and at the end of the article, you can run cmdlets to test the validity of the configuration.

If you want to go into a deep dive about how the Hybrid Authentication works, see the following:
Deep Dive Into Hybrid Authentication – from the MS Exchange Team Blog

Here are some of cmdlets run during this process workflow:

Again, look at both of those links to get a little more detail as to what each cmdlet does and how it sets up OAuth. Here are the two cmdlets used to test OAuth:

10. Enable MRS Proxy for Migration

In order to be able to move mailboxes between Exchange On Premises and Exchange Online, you have to enable the Exchange Web Services Virtual Directory to use the MRSProxy (Microsoft Replication Service proxy). You also have to set your EWS Virtual Directory to use Basic Authentication. You’ll want to do this before running the HCW or else you will receive the following error when the HCW validates the Migration setup and configuration:

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server ‘mail.ldlnet.net’ could not be completed. —> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to ‘https://mail.ldlnet.net/EWS/mrsproxy.svc’ failed. Error details: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. –> The remote server returned an error: (403) Forbidden.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned an error: (403) Forbidden.

Some of the cmdlets run to test Migration and MRS Proxy Settings are as follows:

11. Final HCW Configuration and cleanup.

The HCW runs from final cmdlets to finish up the installation of the Hybrid environment. Here are the cmdlets run:

All this information was found in the setup logs that are in the following directory
C:\Users\%username%\AppData\Roaming\Microsoft\Exchange Hybrid Configuration

REFERENCES
Understanding Federation
Understanding Federated Delegation
Create a Federation Trust
Hybrid deployment prerequisites
Exchange Specific OAuth 2.0 Protocol Specification
Understanding WS-Security
JSON Web Tokens
Using OAuth2 to access Calendar, Contact and Mail API in Office 365 Exchange Online
Configurable token lifetimes in Azure Active Directory (Public Preview)
OAuth Troubleshooting
Principles of Token Validation
Troubleshooting free/busy issues in Exchange hybrid environment
How to configure Exchange Server on-premises to use Hybrid Modern Authentication
Microsoft 365 Messaging Administrator Certification Transition (beta)
Microsoft 365 certification exams
Exchange Server build numbers and release dates

PLEASE LEAVE QUESTIONS, COMMENTS, UPDATES! I WOULD LOVE TO HEAR FROM YOU!

Author Lance LingerfeltPosted on

Update Edge Server Certificate in a Hybrid Exchange Environment

LDLNET LLC Banner
LDLNET LLC – Life In Action! Your Source for Professional IT Services!

At work, our group was updating the Exchange Edge Server certificates and having mail flow problems causing messages to be in the Poison Queue and not transfer to Office365 properly. We finally got the procedure down to where it started working. I wanted to post that procedure here since I had never really worked with Edge Servers in the past. If this post can help you in the future, then “I done good!”

Now, everywhere I had read said that you have to remove and then re-create the Edge Subscription between your Transport Servers and the Edge Servers when changing the certificate.

Here is why:
When we subscribe the edge server, an AD LDS account called the EdgeSync Bootstrap Replication Account (ESBRA) is created. This is created using the default certificate private key of the certificate assigned to SMTP service as default, hence as long as we have that certificate the transport servers will be able to authenticate to the Edge server and replicate the required information to ADAM database.

Now when we install a third party certificate we assign SMTP service to it and overwrite the current certificate, basically we change the default SMTP certificate. So, by doing this, the current Edge Subscription will fail as the Edge server will not be able to decrypt the ESRA account passed on when communicating with the transport servers using the new certificate private key.

So, once you have your new 3rd party certificate, you install it to your edge servers:

Then, you enable the Exchange Certificate to be used for SMTP:

Mail flow will be broken at this point. Since messages were going to the poison queue due to the ESBRA account encryption failing when authenticating with the internal Transport Servers, I had to completely stop transport by disabling the Send Connectors between the internal Transport Servers and the Edge servers from the Transport Server.

The configuration of the Edge Servers were that there were two servers in the Edge Farm. Since one of the servers had not had a proper sync in a while, I decided to remove the recipient database that had been replicated to the failing server when removing the Edge Subscription. The other server, I left the recipient database in place so that we could get one server up and running quickly since transport was stopped at this point.

Here is the command that was run to remove the Edge Subscriptions. This needed to be completed on both the Edge Servers and the corresponding Transport Server:

I then had to create a new Edge Subscription file on each Edge Server to copy to the Transport Server. I already had connectors set so I did not need to recreate those connectors.

I copied the xml files of each Edge Server to the Transport Server and ran the following cmdlet to create the Edge Subscription to the Edge Servers. I then had the Edge Servers Rebooted for good measure before redoing a Full Manual Edge Sync.

I next had to preform a full manual EdgeSync from the transport server to the Edge Servers to assure that the recipient database on the AD LDS instance was up to date and that the send connectors were replicated properly.

I next had to re-run the Hybrid Configuration Wizard so that I could configure the Edge Servers as the transport for Hybrid cloud-bound Messages. Once the Edge Servers were chosen to transport Hybrid cloud-bound messages, I selected the new Edge Certificate so that transport would work properly when re-enabled and O365 would recognize the new certificate for Hybrid messages bound for the cloud.

I next re-enabled the Edge Send Connectors so that mail flow would begin working once the Full Edge Synchronization was completed. You have to let that complete before you can begin mail flow again so that messages won’t be delivered to the Poison Queue.

Mail flow began working. It took about 90 minutes for all the queues to clear properly that had queued messages waiting to transport. Any Poison Queued messages were removed with NDRs sent to the senders.

It was a doozy to say the least. Happy Troubleshooting!
Leave Comments or Questions you may have!

References:
Exchange 2010 Edge Transport Server: Configuring EdgeSync
Mail flow breaks after renewing SSL Certificate on Edge server with Edge Subscription
Start-EdgeSynchronization