Exchange System Mailboxes not being configured cause Exchange Setup to fail

My continuation of the “Installation from HELL” proceeded onward today with our team attempting to install Exchange on another server in the test environment and having it fail when getting to the Mailbox Role portion of the installation.

The error kept saying that the installation was failing due to a “Database is mandatory on UserMailbox”. We had been having many issues with the Schema and RBAC roles which were resolved in my other post by adding the Role Assignments to the schema. I did mention that the environment started settling down and the system mailboxes (Arbitration) along with the Health Mailboxes started functioning. This was actually not the case for the Arbitration mailboxes. I glanced at the following article to see how to manually recreate the Arbitration mailboxes again.

I performed a “Get-Mailbox -Arbitration | fl Name” in Exchange Powershell (similar to the screenshot below) to see if the mailboxes were in fact created. They in fact were not and were giving the error “Database is mandatory for the UserMailbox.”

Image
Verification of Arbitration System Mailboxes existing

So, I tried to do what the original article said to do and enable the mailboxes one by one. I kept getting errors when trying to create the mailboxes. So I began to search the internet for another way to possibly remediate this without having to get too deep into the system.

I found the following article explaining the exact error I was getting during the installation of Exchange. In the article, it said to look at the attributes of the account associated with the Arbitration mailbox to see if the homeMDB attribute had no value:

Image
homeMDB attribute NOT set on Arbitration Mailbox Account

Now, since I was NOT having good luck with either the Exchange Setup nor PowerShell, I had to figure out a way to place the attribute value so that the mailbox would be visible. What I did was this:

  • I opened a User in ADUC with a working mailbox on the needed database.
  • I went to the Attributes Tab and looked up the homeMDB attribute for that user then chose Edit.
  • I copied the entire value from the screen and closed it.
  • I then went to the Arbitration mailbox in question and opened it’s homeMDB attribute.
  • I pasted the value into the Value box and saved it.
Paste the active database value in the homeMDB attribute for the Arbitration Mailbox account

Once completed with remediating the attribute for all the Arbitration mailbox accounts missing the value, I re-ran the cmdlet to verify that the error was not present for any arbitration maibox:

I then uninstalled and re-installed Exchange using setup on the failing server and the installation completed successfully.

This has been an excellent week in training on the value of the setup process for Exchange and also the value of the system accounts and values in relation to Exchange and it working properly.

A POSITIVE OUTLOOK WILL YIELD POSITIVE RESULTS ULTIMATELY!

REFERENCES:
Exchange Install Error Database is mandatory on UserMailbox
Recreate missing arbitration mailboxes

RBAC Role Assignments NOT installed during Exchange Directory Preparation

I had a very interesting installation issue recently when installing Exchange 2019 into a new environment. We ran through all the Exchange Preparation for the root and child domains in the forest as described HERE. The results of those installation procedures showed SUCCESS, but when we started installing Exchange, we ran into issues with the System Mailboxes not being available to complete the Mailbox Role part of the installation. Most of the articles that I found said to re-run the Domain Prep (/preparealldomains) and the AD Prep (/pad). So we did, and managed to get the first server installed somehow.

The reason I said somehow is because when we tried to logon to the EAC, we would get a 400 Bad Request Error and could not logon to the console. Next, we tried PowerShell and was able to load PowerShell, but I noticed that only ~100 cmdlets loaded. I thought that maybe we had to re-create the account mailbox to get it working properly. Problem was, one of the cmdlets that would not load was Disable-Mailbox along with others like Enable-Mailbox and New-Mailbox. It was as if the admin account we were using had no rights to administer Exchange in any way.

Next, we opened the mailbox in OWA. The mailbox came up okay, so I told the admin to change the URL to /ecp to try and get into the admin center. What happened was that the normal user control panel opened instead, showing again that the account did not have permissions.

We checked replication to the child domain and made sure there were not any apparent AD issues present. There were none. I next started reviewing how Exchange uses RBAC (Role Based Access Control) Groups and Role Assignments to grant users access to Exchange Admin Functionality. I read the following article located HERE.

Something told me to go and check the Schema again, so I went to ADSIEdit > Configuration Container > Services > Microsoft Exchange > (Organization Name) > RBAC > Role Assignments

I looked at the list of role assignments in the window as follows:

Small List of RBAC Role Assignments
RBAC Role Assignments Missing Objects

From the picture, you can see that the list is small, which in my experience is not correct. I verified this by going into my own 2019 environment and comparing the number of objects in that folder:

RBAC Assignments Object list with CORRECT Objects Listed

If you notice the list is MUCH longer and has many more objects listed in the container. So, how did Exchange Setup miss this during preparation? That I will find out later, but first I have to remediate this problem.

CAUSE:

If the RBAC roles assignments are not installed to allow an account to have administrative privileges in Exchange, then you cannot administrate Exchange to even make the necessary changes! Especially so if you’ve only installed ONE server in the environment!

REMEDIATION:

Manually repair the installation by running the script that creates these Objects in the Schema during setup.

******DISCLAIMER: Running the following commands in these instructions, running ADSIEdit, and/or making changes to your Schema and Exchange Installation outside the normal setup process is NOT recommended! Microsoft, LDLNET LLC, nor I (Lance Lingerfelt) are responsible for any issues or errors that may arise from using these instructions, period!******

That said, preform the following to regenerate the objects in the Schema:

1) Open Windows PowerShell (not the Exchange Management Shell) on the server that you installed Exchange Server on with the same account you used to install Exchange.

a. If you have UAC enabled, right click Windows PowerShell and click Run as administrator.

2) Run Start-Transcript c:\RBAC.txt and press Enter

a. This will start logging all commands and output you type to a text file.

3) Run Add-PSSnapin *setup and press Enter

a. This adds the setup snap-in which contains the setup cmdlets used by Exchange during install. You may see errors about loading a format data file. You can ignore those errors.
NOTE: DO NOT run any other cmdlets in this snap-in. Doing so could irreparably damage your Exchange installation.

4) Run Install-CannedRbacRoleAssignments -InvocationMode Install -Verbose and press Enter.

a. This cmdlet should create the required role assignments between the role groups and roles that should have been created during setup.

b. Be sure you run with the Verbose switch so we can capture what the cmdlet does.

5) Run Remove-PSSnapin *setup and press Enter

6) Run $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://servername/PowerShell/ -Authentication Kerberos and press Enter

a. Be sure to replace SERVERNAME with the FQDN of your server.

7) Run Import-PSSession $Session and press Enter

a. You should notice that the normal number of cmdlets load (~700)

8) Run Get-ManagementRoleAssignment and press Enter. If you are able to run the cmdlet, then the remedation worked.

9) Run Stop-Transcript and press Enter

The final check is to return to ADSIEdit and check the container and see if all the objects are there. We also were now able to get into EAC as well as saw that the Arbitration mailboxes were populating along with the Health Mailboxes as needed per the installation.
It was very neat to see how running the Add-PSSnapIn cmdlet opened all the scripts from the Exchange Setup and allowed me to manually fix the installation problem by running the cmdlet script that need to perform that task that setup missed or refused to run.

POST MORTEM REVIEW

I am going to look over the installation logs and see where the installation failed and try to find out why it did not run on the subsequent re-installations of the AD Prep and Domain Prep. I will post those finding in this article when I have that available.
Thanks again to my Microsoft and Trimax teammates for your assistance with this. It has helped the customer in more ways than one!

HAPPY TROUBLESHOOTING!
POSITIVE ATTITUDE YIELDS POSITIVE RESULTS!

Why go to Exchange 2019 from 2016 and 2013?

I get asked this a lot in my travels, “What’s the difference with Exchange 2019 and why should we go to it other than the fact that it is the latest version released. I wanted to post my findings from articles that I found to help explain some of the improvements and differences with running Exchange 2019.

Exchange Server 2019 brings a new set of technologies, features, and services to Exchange Server, the messaging platform that provides email, scheduling, and tools for custom collaboration and messaging service applications.

What is NOT in Exchange 2019…

There are some things that have been discontinued in Exchange 2019 which make the decision to go to it important for some companies.

Architecture

FeatureComments and mitigation
Unified Messaging (UM)Unified Messaging has been removed from Exchange 2019. We recommend that Exchange 2019 organizations transition to Skype for Business Cloud Voice Mail.

This is a big deal for some companies as they rely on Unified Messaging to handle their Voice Messaging. There are some articles available to view to assist in transitioning from UM to the new voicemail features with O365. I will post them as I find them in this blog. Keep tuned in for updates!

Here are some other deprecated features:

Client Access server roleThe Client Access server role has been replaced by Client Access services that run on the Mailbox server role. The Mailbox server role now performs all functionality that was previously included with the Client Access server role. For more information about the new Mailbox server role, see Exchange Server architecture.
MAPI/CDO libraryThe MAPI/CDO library has been replaced by Exchange Web Services (EWS), Exchange ActiveSync (EAS), and Representational State Transfer (REST)* APIs. If an application uses the MAPI/CDO library, it needs to move to EWS, EAS, or the REST APIs to communicate with Exchange 2019.

De-emphasized Features

The following features are being de-emphasized in Exchange 2019 and may not be included in future versions of Exchange.

  • Third-party replication APIs
  • RPC over HTTP
  • Database availability group (DAG) support for failover cluster administrative access points (You can have IPLess DAGs now)

What’s new when upgrading to Exchange 2019?

Security

  • Windows Server Core support: Running Exchange on a Windows deployment with less surface area means less attack surface area and fewer components to service.
  • Block external access to Exchange admin center (EAC) and the Exchange Management Shell: You can use Client Access Rules to only allow administration of Exchange from the internal network instead of using complex network and firewall rules.
  • TLS 1.2 is the only version that’s enabled by default: Exchange Server 2019 includes important changes to improve the security of client and server connections. The default configuration for encryption will enable TLS 1.2 only and disable support for older algorithms (namely, DES, 3DES, RC2, RC4 and MD5). It will also configure elliptic curve key exchange algorithms with priority over non-elliptic curve algorithms. In Exchange Server 2016 and later, all cryptography settings are inherited from the configuration specified in the operating system.

Performance

  • Improved search infrastructure: The completely rebuilt search infrastructure for cloud scale and reliability in Exchange Online is now available in Exchange 2019. This new search infrastructure allows for indexing of bigger files, simpler management, and better search performance.
  • Faster, more reliable failovers: The changes to the search architecture result in significantly faster and more reliable failover over between servers.
  • Metacache database: Improvements at the core of Exchange’s database engine enable better overall performance and take advantage of the latest storage hardware, including larger disks and SSDs.
  • Modern hardware support: Exchange now supports up to 256 GB of memory and 48 CPU cores.
  • Dynamic database cache: The information store process employs dynamic memory cache allocation optimizing memory usage to active database usage.

Clients

  • Calendar – Do Not Forward: This is similar to Information Rights Management (IRM) for calendar items without the IRM deployment requirements. Attendees can’t forward the invitation to other people, and only the organizer can invite additional attendees.
  • Calendar – Better Out of Office: Additional options when you won’t be in the office. Key options include: add an event to your calendar that shows you as Away/Out of Office, and a quick option to cancel/decline meetings that will happen while you’re away.
  • Calendar – Remove-CalendarEvents cmdlet: Enables administrators to cancel meetings that were organized by a user that has left the company. Previously, conference rooms or meeting attendees would have these defunct meetings permanently on their calendars.
  • Assign delegate permission via PowerShell: Updates to the Add-FolderPermissions cmdlet so administrators can assign delegate permissions.
  • Email address internationalization (EAI): Email addresses that contain non-English characters can now be routed and delivered natively.

Exchange 2019 architecture

Today, CPU horsepower is significantly less expensive and is no longer a constraining factor. With that constraint lifted, the primary design goal for Exchange 2019 is for simplicity of scale, hardware utilization, and failure isolation. With Exchange 2019, we reduced the number of server roles to two: the Mailbox and Edge Transport server roles.

Unified Messaging (UM) has been removed from Exchange 2019. Other than that, the Mailbox server in Exchange 2019 includes all of the server components from the Exchange 2013 Mailbox and Client Access server roles:

  • Client Access services provide authentication, limited redirection, and proxy services. Client Access services don’t do any data rendering and offer all the usual client access protocols: HTTP, POP and IMAP, and SMTP.
  • Mailbox services include all the traditional server components found in the Exchange 2013 Mailbox server role except Unified Messaging: the backend client access protocols, Transport service, and Mailbox databases. The Mailbox server handles all activity for the active mailboxes on that server.

The Edge Transport role is typically deployed in your perimeter network, outside your internal Active Directory forest, and is designed to minimize the attack surface of your Exchange deployment. By handling all Internet-facing mail flow, it also adds additional layers of message protection and security against viruses and spam, and can apply mail flow rules (also known as transport rules) to control message flow.

For more information about the Exchange 2019 architecture, see Exchange architecture.

Along with the new Mailbox role, Exchange 2019 now allows you to proxy traffic from Exchange 2013 Client Access servers to Exchange 2019 mailboxes. This new flexibility gives you more control in how you move to Exchange 2019 without having to worry about deploying enough front-end capacity to service new Exchange 2019 servers.

MAPI over HTTP

MAPI over HTTP is now the default protocol that Outlook uses to communicate with Exchange. MAPI over HTTP improves the reliability and stability of the Outlook and Exchange connections by moving the transport layer to the industry-standard HTTP model. This allows a higher level of visibility of transport errors and enhanced recoverability. Additional functionality includes support for an explicit pause-and-resume function, which enables supported clients to change networks or resume from hibernation while maintaining the same server context.

Note: MAPI over HTTP isn’t enabled in organizations where the following conditions are both true:

  • You’re installing Exchange 2019 in an organization that already has Exchange 2013 servers installed.
  • MAPI over HTTP wasn’t enabled in Exchange 2013.

While MAPI over HTTP is now the default communication protocol between Outlook and Exchange, clients that don’t support it will fall back to Outlook Anywhere (RPC over HTTP).

Outlook on the Web
(formerly known as Outlook Web App)

Outlook Web App is now known as Outlook on the web, which continues to let users access their Exchange mailbox from almost any web browser.

NOTE: Supported Web browsers for Outlook on the web in Exchange 2019 are Microsoft Edge, Internet Explorer 11, and the most recent versions of Mozilla Firefox, Google Chrome, and Apple Safari.

The former Outlook Web App user interface has been updated and optimized for tablets and smart phones, in addition to desktop and laptop computers. New Exchange 2019 features include:

  • Platform-specific experiences for phones for both iOS and Android.
  • Premium Android experience using Chrome on devices running Android version 4.2 or later.
  • Email improvements, including a new single-line view of the Inbox with an optimized reading pane, archiving, emojis, and the ability to undo mailbox actions like deleting a message or moving a message.
  • Contact linking the ability for users to add contacts from their LinkedIn accounts.
  • Calendar has an updated look and new features, including email reminders for Calendar events, ability to propose a new time in meeting invitations, improved search, and birthday calendars.
  • Search suggestions and refiners for an improved search experience that helps users find the information they want, faster. Search suggestions try to anticipate what the user’s looking for and returns results that might be what the user is looking for. Search refiners will help a user more easily find the information they’re looking for by providing contextually-aware filters. Filters might include date ranges, related senders, and so on.
  • New themes Thirteen new themes with graphic designs.
  • Options for individual mailboxes have been overhauled.
  • Link preview which enables users to paste a link into messages, and Outlook on the web automatically generates a rich preview to give recipients a peek into the contents of the destination. This works with video links as well.
  • Inline video player saves the user time by keeping them in the context of their conversations. An inline preview of a video automatically appears after inserting a video URL.
  • Pins and Flags which allow users to keep essential emails at the top of their inbox (Pins) and mark others for follow-up (Flags). Pins are now folder specific, great for anyone who uses folders to organize their email. Quickly find and manage flagged items with inbox filters or the new Task module, accessible from the app launcher.
  • Performance improvements in a number of areas across Outlook on the web, including creating calendar events, composing, loading messages in the reading pane, popouts, search, startup, and switching folders.
  • New Outlook on the web action pane that allows you to quickly click those actions you most commonly use such as New, Reply all, and Delete. A few new actions have been added as well including Archive, Sweep, and Undo.

Document collaboration
(On-Premises and in O365)

Exchange 2019, along with SharePoint Server 2019 and SharePoint Online, enables Outlook on the web users to link to and share documents that are stored in OneDrive for Business in an on-premises SharePoint server instead of attaching files to messages. Users in an on-premises environment can collaborate on files in the same manner that’s used in Office 365.

When an Exchange 2019 user receives a Word, Excel, or PowerPoint file in an email attachment, and the file is stored in OneDrive for Business or on-premises SharePoint, the user will now have the option of viewing and editing that file in Outlook on the web alongside the message. To do this, you’ll need a separate computer in your on-premises organization that’s running Office Online Server.

Exchange 2019 also brings the following improvements to document collaboration:

  • Saving files to OneDrive for Business.
  • Uploading a file to OneDrive for Business.
  • Most Recently Used lists populated with both local and online files.

Office 365 hybrid and the HCW

The Hybrid Configuration Wizard (HCW) that was included with Exchange 2013 is moving to become a cloud-based application. When you choose to configure a hybrid deployment in Exchange 2019, you’ll be prompted to download and install the wizard as a small app. The wizard will function the same in previous versions of Exchange, with a few new benefits:

  • The wizard can be updated quickly to support changes in the Office 365 service.
  • The wizard can be updated to account for issues detected when customers try to configure a hybrid deployment.
  • Improved troubleshooting and diagnostics to help you resolve issues that you run into when running the wizard.
  • The same wizard will be used by everyone configuring a hybrid deployment who’s running Exchange 2013 or later.

In addition to Hybrid Configuration Wizard improvements, multi-forest hybrid deployments are being simplified with Azure Active Directory Connect (AADConnect). AADConnect introduces management agents that will make it significantly easier to synchronize multiple on-premises Active Directory forests with a single Office 365 tenant.

Exchange ActiveSync clients will be seamlessly redirected to Office 365 when a user’s mailbox is moved to Exchange Online. To support this, ActiveSync clients need to support HTTP 451 redirect. When a client is redirected, the profile on the device is updated with the URL of the Exchange Online service. This means the client will no longer attempt to contact the on-premises Exchange server when trying to find the mailbox.

Secure Messaging, Policy, and Compliance

Data loss prevention

To comply with business standards and industry regulations, organizations need to protect sensitive information and prevent its inadvertent disclosure. Examples of sensitive information that you might want to prevent from leaking outside your organization include credit card numbers, social security numbers, health records, or other personally identifiable information (PII). With a DLP policy and mail flow rules (also known as transport rules) in Exchange 2019, you can now identify, monitor, and protect 80 different types of sensitive information with new conditions and actions:

  • With the new condition Any attachment has these properties, including any of these words, a mail flow rule can match messages where the specified property of the attached Office document contains specified words. This condition makes it easy to integrate your Exchange mail flow rules and DLP policies with SharePoint, Windows Server 2012 R2 File Classification Infrastructure (FCI), or a third-party classification system.
  • With the new action Notify the recipient with a message, a mail flow rule can send a notification to the recipient with the text you specify. For example, you can inform the recipient that the message was rejected by a mail flow rule, or that it was marked as spam and will be delivered to their Junk Email folder.
  • The action Generate incident report and send it to has been updated to enable the notification of multiple recipients by allowing a group address to be configured as the recipient.

In-place Archiving, retention, and eDiscovery

Exchange 2019 includes the following improvements to In-Place Archiving, retention, and eDiscovery to help your organization meet its compliance needs:

  • Public folder support for In-Place eDiscovery and In-Place Hold: Exchange 2019 integrates public folders into the In-Place eDiscovery and Hold workflow. You can use In-Place eDiscovery to search public folders in your organization, and you can put an In-Place Hold on public folders. And similar to placing a mailbox on hold, you can place a query-based and a time-based hold on public folders. Currently, you can only search and place a hold on all public folders. In later releases, you’ll be able to choose specific public folders to search and place on hold. For more information, see Search and place a hold on public folders using In-Place eDiscovery.
  • Compliance Search: Compliance Search is a new eDiscovery search tool in Exchange 2019 with new and improved scaling and performance capabilities. You can use it to search very large numbers of mailboxes in a single search. In fact, there’s no limit on the number of mailboxes that can be included in a single search, so you can search all mailboxes in your organization at once. There’s also no limit on the number of searches that can run at the same time. For In-Place eDiscovery in Exchange 2019, the limits are the same as in Exchange 2013: you can search up to 10,000 mailboxes in a single search and your organization can run a maximum of two In-Place eDiscovery searches at the same time.

Indexing and Search Architecture

In Exchange 2019, the search architecture has been redesigned. It is now based on the same engine as the modern search engines are and is directly on the mailbox in Exchange 2019. There is no content index database attached to the mailbox database as in previous versions of Exchange Server. Previously, search was a synchronous operation that was not very fault-tolerant. The new architecture is asynchronous and decentralized. It distributes the work across multiple servers and keeps retrying if any servers are too busy. This means that we can return results more reliability, and faster.

Another advantage of the new architecture is that search scalability is improved. The number of mailboxes you can search at once using the console has increased from 5k to 10k for both mailboxes and archive mailboxes, allowing you to search a total of 20k mailboxes at the same time.

ENJOY YOUR UPGRADE!
LEARN, DO, LIVE!

REFERENCES:
What is new in Exchange Server
What is discontinued in Exchange Server
Exchange Server TLS Guidance
Exchange Architecture

Exchange 2010 Extended Support will end on October 13th, 2020

I wanted to pass this announcement along to everyone so that they are aware of the support ending for Exchange 2010. I personally have noticed a large number of Exchange 2010 environments starting to show age as the newer Outlook clients are having performance issues with Exchange 2010. If your team has not planned an upgrade to Exchange 2016 (you cannot upgrade directly from Exchange 2010 to 2019), I would advise that your team do so very soon. Exchange 2010 has been a great product for many years, but it is finally time for it to retire and allow the next generation of Messaging Services take the stage.

Formal Announcement:

Exchange 2010 End of Support extended to October 2020

Announced today, and in alignment with Office 2010 and SharePoint 2010, and after investigating and analyzing the deployment state of an extensive number of Exchange customers, Microsoft has decided to move Extended Support date for Exchange Server 2010 from January 14th 2020 to October 13th 2020.
After October 13th 2020, Microsoft will no longer provide technical support for problems that may occur with Exchange 2010 including:

– bug fixes for issues that are discovered and that may impact the stability and usability of the server
– security fixes for vulnerabilities that are discovered and that may make the server vulnerable to security breaches
– and time zone updates

Customer installations of Exchange 2010 will, of course, continue to run after this date; however, due to the changes and potential end of support risks, Microsoft strongly recommends customers migrate from Exchange 2010 as soon as possible.

FAQ’s

  • Can customers upgrade directly to Exchange 2019?
    Customers cannot upgrade directly from Exchange 2010 on-premises to Exchange Server 2019. They may upgrade to Exchange 2013 or 2016 directly from Exchange 2010 and we of course recommend Exchange 2016.
  • Since Exchange 2010 runs on Server 2008 and 2008R2, are those operating systems still supported?
    On January 14, 2020, support for Windows Server 2008 and 2008 R2 will end. That means the end of regular security updates for these Windows customers. Since Exchange Server 2010 runs on top of Windows Server 2008 and Windows Server 2008 R2, it’s important for customers to consider how they will obtain security updates for the underlying operating system. Extended Security Updates for Server 2008 and 2008 R2 are now available for purchase and can be ordered from Microsoft or a Microsoft licensing partner. The delivery of Extended Security Updates (ESU) will begin after the End of Support dates, if and when available. 
  • Does Microsoft support Exchange 2010 on any other Server versions?
    Exchange Server 2010 SP3, with Update Rollup 26 or higher, installed on Windows Server 2012 R2 is supported until October 13, 2020.
  • That didn’t quite answer my question. If a customer calls between January 14 and October 13 2020, and is running Exchange 2010 on Server 2008 or 2008 R2, and does not have an ESU for Windows, can they still be assisted?
    Yes. Per the Lifecycle FAQ.
    If I am running a Microsoft product that is currently supported under the Lifecycle Policy, but my operating system is no longer supported, can I still receive support?
    If the problem is specific to the Microsoft product and it is within the Lifecycle Policy, Microsoft will provide support.
    If the problem is a result of the combination of the operating system and the Microsoft product, the problem will not be supported.
    More simply::
    Exchange 2010 on Server 2008 or 2008 R2: Starting January 14, 2020, provide support until a proven issue is found with the OS. This ends in October 2020.
    Exchange 2010 SP3 RU26+ on Server 2012 R2: We support regardless, but Exchange support still ends in October 2020.

  • Will Microsoft be offering Extended Support Updates (ESU’s) for purchase for Exchange 2010 customers?
    No.
  • What resources are available for customers? 
    – An upcoming Exchange Team blog post, titled “Exchange On-Premises Best Practices for Migrations from 2010 to 2016,” will provide great technical guidance for customers and support agents with their on-premises migrations.  
    – If migrating to Office 365 and Exchange Online, customers may be eligible to use the free Microsoft FastTrack service. FastTrack provides best practices, tools, and resources to make migration to Office 365 and Exchange Online as seamless as possible.
    – For customers that run into any problems during their migration to Office 365 and are not eligible for FastTrack, or if migrating to a newer version of Exchange Server, customers can of course utilize Support or the Exchange Technical Community.
    – Customers may also choose to engage a partner to help.  Microsoft has a great number of partners with deep skills in Exchange, and you can browse a list of Exchange partners at
    https://www.microsoft.com/en-us/solution-providers/home.

HAPPY UPGRADING!
CONTACT ME FOR QUESTIONS CONCERNING UPGRADING YOUR EXCHANGE ENVIRONMENT!

Exchange Server Client Access URL Configuration Script

In my career, I have to be able to be efficient as most of my projects are on a time crunch schedule. Being able to quickly configure Exchange when setting up a server environment is crucial to the success of the project.

While still honing my skills in PowerShell, I was attempting to create my own script to help configure all of the Virtual Directories in one shot rather than go to each setting and configure them manually. It did not go very well, so as I do, I research and find great professionals that do great work in scripting so that I may learn from them.

In doing so, I found Paul Cunningham’s script that performs this. I took the following script and modified it to add the PowerShell Virtual Directory to it as I like to configure that as well.

***YOU CAN REM THE LINES OUT SHOULD YOU NOT WANT TO CONFIGURE THAT DIRECTORY***

Here is my version of the script:

NOTES:

  • PowerShell script to configure the Client Access server URLs for Microsoft Exchange Server 2013/2016. All Client Access server URLs will be set to the same namespace.
  • If you are using separate namespaces for each CAS service this script will not handle that.
  • The script sets Outlook Anywhere to use NTLM with SSL required by default.
  • If you have different auth requirements for Outlook Anywhere use the optional parameters to set those.
  • The script sets PowerShell to use Basic with SSL required by default.
  • If you have different authentication requirements for PowerShell use the optional parameters to set those.
  • PowerShell was added to the settings. Please be sure to REM those lines of code should you NOT want to configure the PowerShell Virtual Directory.

USAGE:

HAPPY SCRIPTING!
POSITIVE ENERGY!
PLEASE COMMENT!

REFERENCES:
Exchange Server Client Access URL Configuration Script
PowerShell Script to Configure Exchange Server Client Access URLs

Installing an ‘IP-less’ Exchange Server 2019 Database Availability Group

Yesterday, I posted on how Exchange now uses the Resilient File System (ReFS) to optimize and protect Exchange critical files. Another layer of protection is using a database availability group (DAG) for redundancy and is a necessary factor when designing an Exchange Enterprise Environment.
In this example, I will walk you through the installation of an Exchange Server 2019 DAG as I configured in my environment. This DAG will contain two Exchange Servers in the same site with a third Windows Server 2019 server being the File Share Witness (FSW).

Two Server Exchange DAG Configuration

For my configuration, I configured two identical Windows Server 2019 VMs (same procs, RAM, vhdx drives, partitions, etc…). I configured the Exchange Data Volume using ReFS and mounted them to the same folder on the C: Drive on each server. This is very important for replication to take place successfully when the databases are added to the DAG.


I next went to the Admin server where the FSW would be hosted and added the Exchange Trusted Subsystem Account to the local Administrators group on that server:

IMPORTANT!
Add the Exchange Trusted Subsystem Account to the Local Administrators Group on the FSW.

NOTE: The reason that this is an ‘IP-less’ DAG is that I’m creating a DAG with no cluster administrative access point (CAAP). The DAG has no IP address of its own, and no computer object in Active Directory. The main implication of this is that backup software that relies on the CAAP or backup operations won’t work. This option of an ‘IP-less’ DAG was first introduced in Exchange Server 2013 SP1/CU4, so by now any decent backup products should support this configuration. But you should always verify this with your backup vendor of choice. Also be aware that this is only supported for DAGs that are running on Windows Server 2012 R2 (or later).

Next, we create the DAG from Exchange PowerShell using the New-DatabaseAvailabilityGroup cmdlet. Now remember that since you are using the ReFS system for your database volumes, you will need to specify the -FileSystem parameter within the cmdlet to assure proper setup and replication of the data files.

Next, we add the Exchange Servers that hold the databases that will be replicated within the DAG:

The DAG will now show the two servers as Operational Member Servers:

The FSW Directory was created on the admin01 server when the DAG was created. We can verify that with the following cmdlet:

Next, we add the databases that we want replicated to the DAG as replicated databases. I want all my Databases on EX01 to replicate to EX02 and vice versa for the EX02 Databases. I want the activation preference to remain on the server that the databases were originally created on so I will use the -ActivationPreference parameter to accomplish that. I will go into more detail on Activation Preference in another post.

Now we verify that the Database Copies are healthy on each replication member using the Get-MailboxDatabaseCopyStatus cmdlet. You will see a Healthy Status on the replicated copies:

POSITIVE ENERGY!
KILL NARCISSISM!
HAPPY TROUBLESHOOTING!

REFERENCES:
Installing an Exchange Server 2016 Database Availability Group

Using the Resilient File System for Exchange Server

In my ongoing effort for becoming more knowledgeable on Exchange Server, I found that the preferred new file system for Exchange Databases and Log files is the ReFS.
ReFS is not that new. Microsoft’s Resilient File System (ReFS) was introduced with Windows Server 2012. ReFS is not a direct replacement for NTFS, and is missing some underlying NTFS features, but is designed to be (as the name suggests) a more resilient file system for extremely large amounts of data.

Support for ReFS with Exchange Server

From Exchange Server 2013 and upwards (which includes Exchange Server 2019 today) Microsoft supports the use of ReFS for Exchange servers, and in fact they now recommend it as the preferred file system for Exchange Server 2019, within the following guidelines.

For Exchange Server 2013:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features must be disabled for the database (.edb) files or the entire volume that hosts database files.
  • Hotfix KB2853418 must be installed.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2013 installation volume, but you can consider using ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2016:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2016 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2019:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.

This means that you should continue to use NTFS for your operating system and Exchange Server 2019 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

Creating an ReFS Formatted Volume

In Windows Server during the New Volume Wizard when you get to the step for configuring File System Settings change the file system from NTFS to ReFS.

exchange-server-refs

NOTE: Using the New Volume Wizard does not give you the option to disable data integrity at the volume level. To set it at the volume level itself use PowerShell when configuring new volumes. I found this out the hard way and am now re-configuring my volumes to disable the Integrity Streams.

I needed to create the mount point to mount the volume to:

I then got a list of my available disks:

In my case, disk 2 was the one I needed to format and change. I had to create a new partition and then format it:

Once formatted, I mount the volume to the Directory created earlier:

NOTE: Partition 1 on a disk is always reserved for system files on the drive volume. So the active partitions will always start at 2.

Lastly, verify that the partition is online and that the Integrity Streams are turned off:

Additional Considerations

When you are deploying an Exchange 2016 or 2019 DAG and using Autoreseed, the disk reclaimer needs to know which file system to use when formatting spare disks. So when, creating a DAG in Exchange PowerShell, make sure to set the -FileSystem parameter. For Exchange Server 2013 DAGs, manually format the spare volumes with ReFS.

More coming soon. I will post how I setup the “IP-less” DAG for my environment and got replication functional for my Exchange Databases.

REFERENCES:
Exchange 2013 storage configuration options
Exchange 2016 Preferred Architecture
Exchange Storage for Insiders: It’s ESE (Ignite video)
ReFS Exchange Server Volumes
Preparing ReFS Volumes for Exchange

Customize your Outlook Web App Logon Page

As many of you are aware, Microsoft provides a default logon page for OWA, the Outlook Web App. Most companies, like myself want to be able to customize that page so that it suites your organization. Here is what my company OWA page looks like:

Customized OWA Logon Page

I have changed the color on the left to match my scheme, replaced the Outlook Logo with my company logo, and added a disclaimer to notify users. Below is the process to do that effectively for your organization.

NOTE: Every time you install an Exchange Cumulative Update (CU) or new version of Exchange Server these modified files will be replaced. Remember to backup your original and changed files to another folder so that you can replace them when you Update or Upgrade or if something goes wrong with the changes.

Customize the color of the Outlook on the web sign-in page

  • Use Notepad to open the file:

%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources\logon.css

  • In the logon.css file, replace the default blue hexidecimal color value #0072c6 with the HTML RGB value that you want to use. You can use the following LINK to choose the color you wish to use.
  • When you’re finished, save and close the file.

Here are the different graphics that can be changed on the OWA logon page and their associated files:

Outlook on the Web sign-in page with element call-outs
ImageFile nameLocationDimensions (width x height in pixels)Bit depth

favicon.ico 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
16 x 16 
32 

olk_logo_white.png 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
128 x 108 
32 

owa_text_blue.png 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
300 x 76 
32 

Sign_in_arrow.png (for left-to-right languages) 
Sign_in_arrow_rtl.png (for right-to-left languages) 
%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\<ExchangeVersion>\themes\resources
22 x 22 
32 
  • Just resize your images to the given dimensions in the table, rename them to the file name, and replace the files in the directory.

Change the disclaimer text for your OWA logon page

Next, we want to add a disclaimer to our logon page. To do that, we need to modify the logon.aspx document in the following directory:

%ExchangeInstallPath%FrontEnd\HttpProxy\owa\auth\logon.aspx

Open the file in Notepad or your favorite HTML editor and search for the text ‘hidden-submit’. When you find the text, you can add your disclaimer text under the div class=”disclaimer” tag as I did in the following example:

Save your logon.aspx file and give your OWA server an IISRESET for good measure. You should be good to logon with the new page from that point on.

HAPPY CONFIGURING!
PLEASE COMMENT!
THANKS FOR YOUR SUPPORT!

References:
Customize the Outlook on the web sign-in, language selection, and error pages in Exchange Server
CUSTOMIZE EXCHANGE 2016 OUTLOOK ON THE WEB SIGN IN PAGE
Customizing Exchange 2016 OWA

Installation of Exchange Server 2019 on Windows Server 2019

I have realized recently that I am an Exchange Messaging Professional, but yet, I have not posted the methodology of how I install an Exchange Server Mailbox Role. So here it is!

Install Windows Server 2019

Exchange Server 2019 requires Windows Server 2019 to run. For my environment, I haven’t necessarily need to follow all the enterprise level design aspects of database numbers to mailbox size ratios, number of servers, front/back end configurations, DAG Implementation, etc… If you want or need to delve into that realm, you can go here. I have need for a single server with only a few databases for a small number of mailboxes, so I am approaching it from that standpoint.

So first, in Hyper-V, I configured my VM with the following specifications:

Processors: 2 procs with 2 cores each – 4 Virtual Processors Total
RAM: 32GB with dynamic memory enabled optional
Drives: 2 .vhdx drives of 120GB each (OS / Exchange Data)
CD: Windows 2019 ISO
Default Settings for the rest of the VM Settings

Next I installed Windows Server 2019 Datacenter with the GUI! You can install it on Server Core if you wish. That information can be found in this link.

I ran through the setup of Windows and installed the OS on my first vhdx drive. I booted up, set the local admin password, and logged in. Once in Windows, I went to the Local Server Settings in Server Manager and configured the following settings:

Set the Date, Time, and Time Zone. (Once in the Domain, this would sync through Group Policy)
Set IE ESC to allow Administrators to have full IE access.
Set Remote Desktop Settings to gain RDP access. (This would be locked down with Group Policy as well once on the Domain)
Set the IP Settings to Static Settings. (DNS Servers, Gateway, WINS, etc…)
Join the server to the Active Directory Domain.
Reboot the VM Server.
Logon to your Domain.
Configure Windows Update Settings. (I have WSUS through Group Policy, this was configured automatically upon reboot)
Download and install all Windows Updates for the server. Then Reboot.
Open Disk Management and configure the secondary vhdx drive to be your Exchange Data Drive.
I configured the drive to be a mounted folder ‘C:\Exchange\Data’ rather than another drive letter as that seems to be the more accepted form of installation for the data drive these days. That is based on the multiple configurations that I have seen for Exchange through experience in Enterprise environments. Again, to each is own and depending on you design specifications, you might want to do that differently.

Next, we need to install the prerequisites for the Exchange Mailbox Server. I have always used practical365.com to get the PowerShell script to install the prerequisites, but couldn’t find the article this time. Great site though! Instead, I got the information and ran the following from an elevated PowerShell Session locally on the server:

As part of the prerequisites you will need to install the following packages onto the server as well:

UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013

Once completed, you can begin the install of Exchange. If this is your first Exchange 2019 Server in your Organization, then you will need to run the following to update the Forest, Schema, and Domain so that Exchange will install properly:

NOTE: If you run into Prerequisite issues with the installation due to a “pending reboot”, check out my blog post for information on remediation of that issue.

Now that the environment is prepared for Exchange, you can actually begin the installation. I wanted to make my default database and logs folder to be on the Exchange Data volume that I created, so I included those settings in the setup command. Please look at the reference to the setup.exe switches for more information on that. Here is the command:

Setup should go through the installation via the PowerShell window and complete successfully. Reboot the Exchange Server, then you can then logon to the Exchange Admin Center and begin the process of configuration of how you need to integrate the Mailbox Server into your Server Farm. That configuration is for a later post.

PLEASE CHECK BACK FOR UPDATES!
PLEASE COMMENT!

References:
UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013
Install Exchange Server 2019 on Windows Server 2019 Core
Exchange Server Design Planning
Use unattended mode in Exchange Setup
Practical365 on Exchange 2019

Exchange Hybrid Configuration Wizard Link

Wanted to do a quick post as I was working on my Hybrid Exchange Environment. I was unable to get the HCW to download and start from the Exchange Control Panel with the link provided on the page. This has happened to me for a while, so I went online and found a link that would work that could be downloaded and reused to open the HCW:

Hybrid Configuration Wizard Link

HOPE THIS HELPS!
LET ME HAVE KNOWLEDGE SHOULD THE LINK CHANGE!

References:
HYBRID CONFIGURATION WIZARD WON’T START ON WINDOWS 2016

Exchange Setup Repeatedly Says ‘A Restart from a Previous Installation is Pending’

I have had this issue with EVERY upgrade that I have ever attempted for Exchange Server from 2013 through 2019 CU1. You go to run the setup program and during the prerequisite checks, setup stops. The error listed is:

A restart from a previous installation is pending. Please restart the system and rerun setup.

During the prerequisite checks, Exchange Setup looks in the registry at the following keys:

  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Updates\UpdateExeVolatile
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations

Nine times out of ten, a restart does NOT remediate this error. In order for setup to continue properly, you must do the following:

  • Open regedit: Start > Run > regedit.exe
  • Set the HKLM\SOFTWARE\Microsoft\Updates\UpdateExeVolatile key value to 0 or delete it if present. <– This one is usually NOT present.
  • Delete the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations key.
  • Re-run Setup.

You should now be able to run setup and upgrade your Exchange Server.

PLEASE COMMENT!
HAPPY TROUBLESHOOTING!

References:
A Restart From Another Installation Is Pending
Exchange Setup Fails – A Restart From Another Installation Is Pending
Microsoft Document – A Restart From Another Installation Is Pending