COVID-19 Update from LDLNET LLC

Message of Support

It has always been my goal to provide your company with sthe best IT solutions and I will continue to do so as effectively as I can. 

I am aware that the recent spread of COVID-19 may have you concerned. At LDLNET LLC, I take the safety and well-being of my present and future customers seriously.

What I Am Doing

To further prevent the spread of COVID-19, I am committed to enacting best practices laid out by the Center for Disease Control (CDC). To ensure the health and safety of your offices, I clean, sanitize and disinfect my equipment with EPA approved sanitizer and disinfectant between appointments. I can also provide 100% remote work to you company provided that I have secure access.

How I Can Help 

With precautions taken, I will continue to stay open for business. I will continue to provide the best IT Solutions for your business that I can. Please feel free to contact me to set up an appointment for your next IT project!

I sincerely hope that you and your family are in good health. As our community works towards a solution, please take your own precautionary measures to reduce your own risk of illness. The simple act of washing your hands can act as an effective barrier against germs and bacteria. I want everyone in our community to stay safe.

If you want to schedule a time to talk, give me a call today at (704) 222-2366 to schedule an appointment or visit the website http://store.ldlnet.net for information. 

Sincerely,
Lance Lingerfelt 
Owner/Operator 
LDLNET LLC

Remember to clean and disinfect high-touch surfaces daily in all common areas.

Hand washing is one of the best ways to protect you and your family from getting sick.

If you show symptoms or are sick, it is recommended that you stay home.

STEPS TO DECOMMISSIONING YOUR EXCHANGE 2010 ON-PREMISES ENVIRONMENT

This was a great article released by the Exchange Team Blog today, and as I have been dealing with MANY customers still having Exchange 2010, I wanted to have this available for quick review! It has great links and steps to consider when finally getting off Exchange 2010.

Best practices when decommissioning Exchange 2010

As many of you know from the previous post regarding Exchange On-Premises Best Practices for Migrations from 2010 to 2016 the end of support for Exchange 2010 is quickly approaching. We’ve created this post to cover the best practices for decommissioning an Exchange 2010 environment after the migration has completed.

Uninstalling Exchange 2010 is as easy as running Setup and selecting to remove the server roles, but there are prerequisites to removing the roles and legacy items left over, which should be removed.

This post is intended to provide best practices to plan for and complete the Exchange 2010 decommission. Please note that since there are many different types of deployments and configurations it is difficult to cover all scenarios, but many of the common steps are included here. Please plan the decommission process carefully.

As a general statement, here are some things that we want to caution against:

  • Do not reuse Exchange 2010 server names (until they have been fully decommissioned).
  • Do not reuse Exchange 2010 server IP addresses (until they have been fully decommissioned).

This post assumes that your organization is maintaining some Exchange presence on-premises, whether Exchange 2013 or 2016 (we do not mention Exchange 2019 in this post because it cannot coexist with Exchange 2010). If your organization has moved all mailboxes to Office 365 and is in a Hybrid environment, we are assuming you will maintain an Exchange footprint per Scenario 2 in How and when to decommission your on-premises Exchange servers in a hybrid deployment.

Preparing for Soft Shut Down

Once you’ve completed the migration from Exchange 2010 to, let’s say, Exchange 2016, you should prepare the 2010 environment prior to decommissioning the servers. The following steps to consider are separated into server roles when preparing for a soft shut down and preparing for the removal of server roles.

Client Access (CAS) Role

Check Server FQDNs

Review all namespaces (e.g. DNS records and load balanced virtual IP addresses) used for client connectivity and ensure they are routing to the 2016 environment. These are all the names that are published for Outlook Anywhere, AutoDiscover, and all Exchange Virtual Directories.

Tip: Verify that all clients such as ActiveSync, Outlook, EWS, OWA, OAB, POP3/IMAP, and Autodiscover are no longer connecting to the legacy Exchange servers. Verification of this can be done by reviewing the servers’ IIS Logs with Log Parser Studio (LPS). LPS is a GUI for Log Parser 2.2 and it greatly reduces the complexity of parsing logs. LPS can parse large sets of logs concurrently (we have tested with total log sizes of >60GB). Please refer to the following blog post with tips and information on using LPS.

Check SCPs

Make sure that the Service Connection Point (SCP) is moved to Exchange 2016 as discussed in the Exchange On-Premises Best Practices for Migrations from 2010 to 2016 post under the Configure Autodiscover SCP for Internal Clients section.

If present, ensure that if the AutoDiscoverServiceInternalURI routes to an Exchange 2016 endpoint. You can also remove this value by setting the AutoDiscoverServiceInternalURI to $Null.

Hub Transport Role

Follow the items below to review all mail flow connectors. We will not be removing connectors themselves, simply auditing to ensure that the server is ready to be decommissioned.

Review the Send Connectors

Review the send connectors and ensure that the legacy servers have been removed and Exchange 2016 servers have been added. Most organizations only permit outbound network traffic on port 25 to a small number of IP addresses, so you may also need to review the outbound network configuration.

Review the Receive Connectors

Review the receive connectors on legacy servers and ensure they are recreated on your Exchange 2016 servers (e.g. SMTP relay; anonymous relay; partner, etc.). Review all namespaces (e.g. DNS records and load balanced virtual IP addresses) used for inbound mail routing and ensure they are terminating against the Exchange 2016 environment. If your legacy Exchange servers have any custom, third-party, or foreign connectors installed (for example, with fax services), ensure that they can be reinstalled on 2016 Exchange servers.

Tip: Check the SMTP logs to see if any outside systems are still sending SMTP traffic to the servers via hard coded names or IP addresses. To enable logging, review Configure Protocol Logging. Also, ensure we have “time coverage” for any apps relaying weekly/monthly emails that may not be caught in a small sample size of SMTP Protocol logs. There is a great script available here that can help find any applications that may be relaying off your legacy environment.

In general, the decommissioning process is a great time to audit your mail flow configuration to ensure that all the connectors are properly configured and secured. Maybe it’s time to get rid of any of those Anonymous Relay connectors that may be in use in your environment. Or, if Hybrid, possibly relay against Office 365.

Transport Rules

Exchange 2010 base transport rules are held in a different AD container than Exchange 2013 and newer rules. When installing Exchange 2016 in your environment it will import those Exchange 2010 based rules. However, any changes to Exchange 2010 rules after a later version of Exchange is installed must also be applied to your Exchange 2016 rules. This is further explained here under section Coexistence with Exchange 2010.

Run the following command to get all your Exchange Transport Rules. Must be run on Exchange 2016 to see all rules.

Compare the rules with RuleVersion of 14.X.X.X to those with 15.1.X.X. If any Exchange 2010 rules don’t exist on Exchange 2016, they must be created. Also review all settings of each Exchange 2010 rule and replicate them to Exchange 2016.  

Mailbox Role

Identity and move all Exchange 2010 mailboxes to Exchange 2016

Decommissioning Exchange 2010 cannot be initiated until all mailboxes have been moved to Exchange 2016. As an example, we cannot decommission Exchange 2010 Hub Transport servers completely until all of the mailboxes are moved off the legacy platform, this is due to how Delivery Groups are handled.

We encourage using the newest Exchange platform to process any move requests. If moving to Exchange 2016, move all mailboxes via Exchange 2016. Also, ensure that once all moves are completed, and that all associated Move Requests are removed as well. Any lingering move requests or mailboxes will prevent uninstallation of Exchange 2010.

To move all user mailboxes, run the following command to identify the mailboxes, and then plan to move them to the new platform.

Tip: Ensure that Archives are included with “Get-Mailbox -Archive” if you used Exchange Archives in 2010. Also, do not forget about your Discovery Search mailboxes – these can be found with: Get-Mailbox -Filter { RecipientTypeDetails -eq “DiscoveryMailbox”}. These will need to be moved (if they haven’t yet already), to Exchange 2016 as well.

Identify and Move Arbitration Mailboxes to Exchange 2016

It’s necessary to move the arbitration mailboxes from Exchange 2010 to 2016 for many Exchange Services to work properly, including the Exchange Admin Center (EAC). This is typically executed when Exchange 2016 is first installed, however, if that was missed, we will ensure that is handled now. The process to move is defined at: Move the Exchange 2010 system mailbox to Exchange 2013+. To verify which system mailboxes are located on 2010, use PowerShell on your Exchange 2010 server with the following example:

Note: If any mailboxes are present, move them to an Exchange 2016 database.

OAB Generation

Installing first Exchange Server 2013+ into Exchange 2010 organization creates a new OAB. It also marks the new OAB as default. The Exchange 2010 OAB is not used by Exchange 2013+ servers so moving the OAB is not necessary. Move the OAB to another Exchange 2010 server, if you are removing an Exchange 2010 server that’s currently hosting the OAB, and there are other Exchange 2010 servers in the org. If you are removing the last Exchange 2010 server in the org, remove the OAB.

Migrate All Legacy Public Folders

Verify that all the public folders have been migrated to Exchange OnlineOffice 365 Groups, or Exchange Modern public folders.

Mail Enabled Public Folders (MEPF) consideration

If the following is true:

  • Exchange Server 2010 public folders are migrated to Exchange Online
  • Exchange Server 2013/2016 was introduced on-premises
  • MEPF’s are still used on-premises to send emails to Exchange Online

In that case, you may need to run the SetMailPublicFolderExternalAddress.ps1 script to ensure Exchange 2013+ servers can continue sending emails to Exchange Online MEPFs.

Decommission the Database Availability Group (DAG)

Assuming best practices were followed for the Exchange 2010 environment, we will have a DAG for HA/DR capabilities. Now that all mailboxes have been removed from the 2010 environment, we are ready to tear down this DAG to move forward with decommissioning Exchange 2010.

Remove Database Availability Group (DAG) Copies

First, we start with the copies. For every mailbox database copy in the environment hosted on Exchange 2010, we will need to remove the Mailbox Database Copy. This can be done via the UI, or via PowerShell:

NOTE: Removing the copy will not remove the actual .edb database file from the Server.

Remove All Nodes from Database Availability Group(s) (DAG)

For each Exchange 2010 server in the environment, we will need to remove the individual server from the DAG. This is evicting the server from the cluster. This can be done via the UI, or through PowerShell.

Remove DAGs

Lastly, once the Database copies are removed, and the servers are evicted from the cluster, the last thing is to finally remove the DAG from the environment. This can be done with the following PowerShell command:

Tip: If you have an even-membered DAG, and leveraged a File Share Witness, don’t forget to decommission the file share witness that was used for the Exchange 2010 DAG.

Unified Messaging Role

Configuration steps are required to move Exchange 2010 UM to Exchange 2016 servers. The following link can be used to guide through removal of UM from Exchange 2010. If moving to a third-party UM solution, remove the UM components to allow un-installation of the UM role.

Edge Role

If you have an Edge server, you will need to install Exchange 2016 Edge and recreate the Edge Subscription on the E2016 server. This is further documented here.

Other

As mentioned in the beginning of the document, due to so many different types of deployments and configurations, it’s difficult to cover all scenarios however it’s recommended to check any other possible scenarios that apply to your environment.

Third Party Applications

Make a list of applications that may be using Exchange 2010 (e.g. EWS, mail transport, database-aware) and make sure to configure these applications to start using Exchange 2016 infrastructure.

Shut-Down Exchange 2010 Servers

Test shutting down the Exchange servers for a few days to a few weeks to see if there are any issues. You are auditing for any applications that are trying to connect to the Exchange 2010 servers or trying to send email through the Exchange 2010 servers.  Enabling protocol logging on the Hub Transport roles prior to shutting down the servers is an option. That way if any mail is processing through these servers, upon restart, the logging will begin immediately.  If applications or servers are trying to connect you can remediate those or power on the Exchange 2010 servers until remediation can happen.

Tip: Check Active Directory DNS Zone settings to see if DNS Scavenging is enabled.  If this is enabled, the DNS record could become stale during the shutdown time frame and cause DNS issues for the Exchange 2010 server.

Preparing for Removal of Server Roles

As you begin the process of removing servers, you should go through the list below and ensure you have everything tested and ready to go.

CAS

Remove CAS Arrays

Remove Any Exchange 2010 Client Access Arrays from Active Directory and DNS. Refer to the following document to remove the Client Access Array object with Shell using the following example:

Be sure to also remove any references in DNS to the CAS Array Name.

Remove Unused 2010 ASAs

If you followed either the Best practices for Migrations blog or the Coexistence with Kerberos blog, we recommend that any old alternate service accounts (ASAs) used for E2010 be removed. If you are using a different namespace than Exchange 2016, please verify old SPNs are also removed.

Remove Exchange 2010 OAB

Use the following command to remove Exchange 2010 OAB:

Remove Mailbox Databases

Now that all mailboxes are migrated from the Exchange 2010 platform, and the DAG is properly removed, we will want to decommission any leftover databases from the Exchange 2010 environment. To remove all Exchange 2010 databases, review the output of the following, and remove individually:

And then remove the database with:

NOTE: If there are any mailboxes currently residing on the database, we will not let you remove the database, it will fail with the following error:

e2010decom1.jpg
Remove Legacy Public Folders

If you chose not to migrate public folders, refer to the following document to remove public folders with either EMC or Shell using the following example:

Remove Legacy Public Folder Databases

Refer to the following document to remove the public folder databases with PowerShell using the following example:

Tip: Remember the .edb files linger after the above is done. Feel free to delete, backup, or do with these as you please.

Uninstall Exchange 2010

It’s recommended to uninstall in the following order: CAS, Hub, UM (if any), then Mailbox.  

Starting the Uninstall Process

When you begin the uninstall process, close EMC, EMS, and any additional programs that could delay uninstall process (i.e. programs using .NET assemblies; antivirus and backup agents are examples). You can either run Exchange 2010 Setup.exe or navigate to Control Panel to modify or remove Exchange 2010 (either server roles or the entire installation). Specific steps are discussed in Modify or Remove Exchange 2010.

Tip: Exchange will protect itself! If you properly uninstall via Add/Remove Programs, it will ensure that it is ready to be uninstalled via Readiness Checks! If all the above prep work is completed before hand, it should uninstall just fine.

After Uninstall of Exchange 2010

After uninstalling Exchange there will be some general “housekeeping” tasks. These may vary depending on the steps taken during your upgrade and depending on your organization’s operational requirements.

Examples include:

  • Removing the legacy Exchange computer accounts from AD (including the DAG’s Cluster Name Object and any Kerberos ASA object).
  • Removing the legacy Exchange name records from DNS (including the DAG’s Cluster Name Object and any Kerberos ASA object).
  • Ensure the folder on the DAG file share witness (FSW) servers were successfully removed, possibly removing Exchange’s rights on the server if it isn’t serving double duty for Exchange 2016.
  • Removing old load balanced IP addresses and routes from your network load balancer.
  • Remove old firewall rules that open ports to Exchange 2010 environment.
  • Removing and disposing of the legacy Exchange environment’s physical equipment.
  • Deleting of the legacy Exchange environment’s virtual machines.

Conclusion

With the uninstall of the last server, hopefully Exchange 2010 treated your organization well. The Exchange product team takes great pride of the success of the platform and hope that you see the same success with Exchange 2016 (or Exchange Online!). Messaging sure has come a long way since it was released way back in 2009.

REFERENCES
Exchange Team Blog article on Decommissioning Exchange 2010 On-Premises

CHECK FOR CONTINUED UPDATES!
THANKS FOR STOPPING BY!

Exchange Server Quarterly Updates March 2020

Released: March 2020 Quarterly Exchange Updates

Today Microsoft is announcing the availability of quarterly servicing cumulative updates for Exchange Server 2016 and 2019. These updates include fixes for customer reported issues as well as all previously released security updates. 

Personal Note: I was recently involved in a layoff at Microsoft in the Vendor PFE world. I am currently looking for new engagements.

Calculator Updates

This quarterly Exchange release includes an important update to the Exchange 2019 Sizing Calculator.  We’ve made improvements to the logic to detect whether a design is bound by mailbox size (capacity) or throughput (IOPs) which affects the maximum number of mailboxes a database will support.  Previous versions of the calculator produced incorrect results in some situations.

The Exchange team highly recommends using calculator version 10.4, included with the March 2020 quarterly CU release, to size Exchange Server 2019 deployments.

MCDB Configuration Issues

Cumulative Update 5 for Exchange Server 2019 also fixes an issue that can happen when you use the Manage-MetaCacheDatabase.ps1 script to enable MetaCacheDatabase (MCDB).

This issue occurred because of a change in behavior in Windows Server 2019 that caused Get-Disk to return all uninitialized discs within the Database Availability Group (DAG) or cluster. The script then incorrectly tried to format an SSD on another DAG member. We documented a workaround for CU4 here, but we’ve fixed it in CU5.

Online Mode Search Issues

Cumulative Update 5 for Exchange Server 2019 is also required to fix a known issue with partial word searches when the client is using Outlook in online mode.

Release Details

The KB articles that describe the fixes in each release and product downloads are available as follows:

Additional Information

Microsoft recommends all customers test the deployment of any update in their lab environment to determine the proper installation process for your production environment. For information on extending the schema and configuring Active Directory, please review the appropriate documentation.

Also, to prevent installation issues you should ensure that the Windows PowerShell Script Execution Policy is set to “Unrestricted” on the server being upgraded or installed. To verify the policy settings, run the Get-ExecutionPolicy cmdlet from PowerShell on the machine being upgraded. If the policies are NOT set to Unrestricted you should use the resolution steps in KB981474 to adjust the settings.

Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, or who are using Exchange Online Archiving (EOA) with their on-premises Exchange deployment are required to deploy the currently supported cumulative update for the product version in use, e.g.,

2013 Cumulative Update 23
2016 Cumulative Update 16 or 15
2019 Cumulative Update 5 or 4.

For the latest information on Exchange Server and product announcements please see: 
What’s New in Exchange Server and Exchange Server Release Notes.

I AM STILL CURRENTLY LOOKING FOR A NEW PROJECT OR ASSIGNMENT!
THANKS FOR READING!

Exchange Server Security Update KB4540123 fails with 0x80070643

PLEASE READ THE ENTIRE POST

I had a failure on one of my two Exchange 2019 CU4 servers when installing the Security Update for them:

Exchange Server Security Update KB4540123 fails with 0x80070643

I could not restart the install as it would fail when getting to the services stoppage part of the installation. I saw this error in the ServiceControl.log file in the C:\ExchangeSetupLogs Directory

I had searched around based on the error code 0x80070643 and found these answers that some had used to get the installation to work.

LINK HERE

I downloaded the .msp file to install manually and read the answers on the web-page. It was said that there was an issue with the ServiceControl.ps1 file that is in the Exchange Server BIN directory when running Patch with the Verbose logging enabled:

I kept reading and digging into the fix for the file. It was said to modify a number of lines in the ServiceControl.ps1 script:

Once I made those changes. I renamed the original ServiceContol.ps1 to a .old file and saved this modified file to the BIN directory. I was then able to successfully run the Security Update.

BUT WHY DID THIS SERVER FAIL AND NOT THE OTHER?!?

Both are CU4, but the server that failed had been upgraded from CU1 where the one that did NOT fail was a clean installation of CU4. So, I checked the ServiceConrol.ps1 file on both servers:

Older Exchange Install
Date of ServiceControl.ps1 is 1/1/2020
CU4 Clean Installation
Date of ServiceControl.ps1 is 2/3/2020

I have not tested this, but maybe if I had copied the ServiceControl.ps1 file from the CU4 clean installation to the original install, the script might have worked since the creation dates are different and I have no other version information to go on. I will verify this though. For now, the changes to the script allowed me to successfully install the Security Update Successfully.

NOTE: All the Exchange and IIS Services were in a Startup Mode: Disabled state and I had to reset them ALL to Automatic. Once that was completed and the server rebooted, Exchange was returned to normal state. ****Also, just to be safe, I ran a great script that assures the server is out of maintenance mode. You can get the script HERE. You can also get the script to put the server into maintenance mode. You can get the script HERE. These scripts will work on Exchange 2013 and above servers.

UPDATE / 3/12/2020 2:39 AM EST

Something was still wrong with the server. ActiveSync and PAM started breaking. I started having all sorts of authentication problems. I decided to restore the server from backup from Tuesday. I forgot to backup the ServiceControl.ps1 file that modified, but I moved the one from the successful installation to the restored server and am currently running the update. I will see if it works and send the information to you.

UPDATE / 3/12/2020 3:45 AM EST

The restore worked great and placing the ServiceControl.ps1 file in the BIN directory on the prior failed Exchange Server did allow for the installation to complete successfully. I have tested ActiveSync and Authentication which is now functioning properly. Hooray!

SEND ME YOUR IDEAS/ FOR POSTS!
HAPPY TROUBLESHOOTING!

REFERENCES:
Exchange Server Security Update fails with 0x80070643
Exchange Maintenance Mode Script (Start)
Exchange Maintenance Mode Script (Stop)
Exchange Server 2019 CU4 Security Update

Exchange 2016 Deployment MUST READ Documentation

In my role as a PFE for Microsoft, I have been going through many deployments of Exchange 2016 from Exchange 2010 due to the end of life deadline for Exchange 2010. I am actually in the middle of three enterprise level deployments this month.

Because of this, I wanted to provide some MUST READ links with consideration to the Exchange 2016 Deployment process. Please click on the links below, and they will take you to the documentation needed when preparing for a Exchange 2016 migration and deployment from Exchange 2010.

IMPORTANT READ:
Exchange On-Premises Best Practices for migration from Exchange 2010 to Exchange 2016 (Lots of important links and articles within this document!)

Other relevant articles:
Exchange 2016 System Requirements
Exchange Deployment Site Consideration
What changes in AD when Exchange 2016 is installed
Exchange 2016 Schema Changes to AD
Exchange Server Virtualization
Exchange Server Preferred Architecture
Load Balancing in Exchange Server
Load Balancing Exchange 2016 In Depth

Hybrid Considerations:
Decommissioning Exchange 2010 servers in a Hybrid Deployment
How and when to decommission On-Prem Exchange Hybrid Servers
Hybrid Deployment Prerequisites

I will add more articles as they become relevant in my experiences with my customers and feel they could be relevant here as well. If you have a suggestion for a link that should be considered added to this post, feel free to leave a comment!

THANKS AGAIN FOR READING!
SUGGEST A LINK!
GOOD LUCK ON YOUR DEPLOYMENT!

Adding Windows Capability to Server Core to add features needed for Application Compatibility

I’ve been working on installing Windows Server 2019 Core into my network to be able to look at new features for Windows Administration and learning how Server Core works. I was able to install a virtual machine with Server Core and get it activated. I then wanted to place my custom PowerShell script for loading PowerShell into the Server Core Environment.

So, I added the Server Core Server to the Windows Admin Center and copied my custom scripts for PowerShell into the proper directory:

Windows Admin Center
Windows Admin Center

I then logged on remotely to the server and started PowerShell. When I did that, I got this error with the script load:

Error that IE First Run has not been completed
Error that IE First Run has not been completed

At first, I tried using the -UseBasicParsing as a switch to see if that would repair the issue in the script. It did not because, IE is not installed by default on the default installation of Server Core. That is so there is less of a footprint that can be attacked by a hacker. I needed this installed though so that the Invoke-WebRequest cmdlet would load my script parameters properly.

I started looking for answers to how to install IE onto the Server Core box and found the following article. I had to run the Add-WindowsCapability cmdlet on the server to install the optional components. When I did, I received an error:

Error when adding the Windows Capability
Error when adding the Windows Capability

So I found out that there is a block that WSUS does keeping the cmdlet from going to the online source to download the software package and producing this error. After researching, I found this article. I setup a Group Policy to make sure this setting is propagated to my Server Core machine. I also setup in the same policy the ability to turn off the First-Run for IE so that you do not get that message and have to open IE to “set it up”

Group Policy Setting with Path to Templates Specified
Group Policy Setting with Path to Templates Specified

I then ran a gpupdate /force on the Server and was able to download the components for IE and App Compatibility.

Successful Installation of Windows Capability
Successful Installation of Windows Capability

I then rebooted the server and now my PowerShell loads successfully:

Successful PowerShell Load
Successful PowerShell Load

I learned a few different new things here and was able to get Server Core working more the way that I like it. I will keep posting updates when I run into issues with this type of installation. I would definitely give the Windows Admin Center a try as it has more robust features than Server Manager has, especially for Server 2019 and Server Core.

CONQUER THE UNCOMFORTABLE TO GROW!
POSITIVE ATTITUDE ABIDES!

REFERENCES:
RSAT Tools Installation Error 0x800f0954 – Windows 10 1809
Server Core App Compatibility Feature on Demand (FOD)
“Set Up Internet Explorer 11” Bypass with GPO or Registry

How to address Federation Trust issues in Hybrid Configuration Wizard (HCW)

During my time as a PFE for Microsoft, I have encounted many issues with Federation in a Hybrid Exchange Deployment. Recently, the following support announcement came out and I wanted to share as I hope this can help others that may be having issues out there.

One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).

Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.

Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):

Example of Hybrid Federation

Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) – as this is the supported tool to configure hybrid deployments.
There are 2 flavors of hybrid configurations:
Classic hybrid
Modern hybrid

At this time, each of those supports the following hybrid modes:

  • Full
  • Minimal (which further breaks down into…)
    • Express (a one-time sync)
    • “Actual minimal”

A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.

As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.

HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.

When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed:

Set-FederationOrganizationIdentifier
or
Add-FederatedDomain (but can be other cmdlets as well).

Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.

In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.

For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.

Example:

If the Exchange Server version is Exchange 2013/2016 and the above commands didn’t show more details on the error, we can also try the following:

  • Open regular Windows PowerShell (blue background) on the Exchange Server 2013/2016
  • Run command: add-pssnapin exchange
  • Run command that gave error in HCW and add a Verbose switch

Example:

Once you’ve gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).

Common Errors with Remediation Steps

  • Federation trust fails with “Object reference not set to an instance of an object”

This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.

Full error in the HCW log:

Resolution: Install the latest CU for Exchange 2016


  • Federation fails with “Proof of domain ownership has failed”

Full error in the HCW log:

Resolution:

• Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName
• See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://www.whatsmydns.net/ put your domain in hostnames, type=txt, Nameservers – Authoritative

You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).


  • Federation fails with “An unexpected error occurred on a receive” or “An unexpected error occurred on a send.”

Full error in the HCW log:

Resolution:

Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account).

In this example, “Windows Live” is actually this exact URL: https://domains.live.com/service/managedelegation2.asmx

From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx

Verify the machine/system account can access these Microsoft Federation Gateway URLs:

For a complete list of O365 URL & IP addresses, see these articles:

Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using “Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80”. Notice such proxy can’t require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).

You can also set the proxy using netsh as well.

set proxy proxy-server=”http=myproxy;https=sproxy:88″ bypass-list=”*.contoso.com” 

In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with “Could not establish trust relationship for the SSL/TLS secure channel.” If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.

REFERENCE:
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)
Firewall Considerations for Federated Delegation 

Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.


  • There is no specific error / exception, in HCW log you would see it stops without any specific error.

Full error in the HCW log:

Resolution:

Look for orphaned federation trust:

Get-FederatedOrganizationIdentifier | FL

or

in HCW log if you see something with “DEL“: “contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>”.

Solution is to remove the orphaned federation trust and re-run HCW.

Reference here.

NOTE: as a first step, you can try to run the command remove-federateddomain with the switch -Force. Also, you don’t need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)


  • Federation Trust fails with “InternalError InternalError: Internal error.”.”.””

Full error in the HCW log:

Resolution:

Open request with Microsoft Support or check if any Service Incident is published. Please see this for more information.


  • Federation trust fails with “1007 Access Denied”

Full error in the HCW log:

Resolution:

“1007 Access Denied” error is usually when we have issues with:

  1. Windows Time on the Exchange Server. See this article or this article.
  2. Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.

If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.

As an example, from one HCW log, there seems to be this federation trust certificate expired on 05/13/2019:


  • Federation trust fails with “Federation Certificate cannot be found”

Full error in the HCW log:

Resolution:

Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run the HCW to re-create it automatically.

KEEP TROUBLESHOOTING!
REMAIN VIGILANT USING FOCUSED INTENT, NEVER EMOTIONALIZING (FINE)!

REFERENCE:
How To Address Federation Trust Issues using the Hybrid Configuration Wizard

Fixed | WSUS 2016/2019 | Error Code 0x8024401c | Windows 10 nor Windows Server 2016 reporting to WSUS

Problem:

I had recently had this error in WSUS where my Windows Server 2016 servers would NOT report into the WSUS Server. I would get an error stating 0x8024401c when manually performing a report now to the WSUS Server using:

Error from Windows Update on affected server

Solution:

Go to IIS Manager on the WSUS Server

Goto Advanced Settings of  WsusPool.

Make sure following settings are present/configured on the Pool, if not change it to below:

Make sure, the WSUS Entry in the Registry is having fully qualified domain name of WSUS Server.

NOTE: If you have Group Policy managing the WSUS Settings, then make sure you change the settings in the WSUS Policy to use the FQDN of the WSUS Server and run a gpupdate /force on the clients.

[image%5B2%5D]
Should be set to FQDN of your WSUS Server
i.e. “http://wsus.domain.com:8530”

Stop IIS on the WSUS Server

Edit the web.config located at following location on WSUS Server:

Replace the following lines in the config file and save in the same directory:

Restart IIS on the WSUS Server

Try updating the clients again. They should be able to report and update successfully.

HAPPY TROUBLESHOOTING!
POSITIVE OUTCOMES ARISE FROM POSITIVE ATTITUDES!

REFERENCES:
Fixed | WSUS 2016 | Error Code 0x8024401c | Windows 10 | Windows Server 2016

Exchange 2019 Setup Prerequisite Check fails for .NET 4.8 Framework in CU4 on Windows builds 1909 and 1903

Symptoms

When you deploy or upgrade to Microsoft Exchange 2019 Cumulative Update 4 (CU4) on Microsoft Windows Server 2019 or Windows 10 (Management Tools only) builds 1909 or 1903, the system prerequisites check fails, and you receive the following error message:

“This computer requires .NET Framework 4.8 (https://support.microsoft.com/kb/4503548).”

By default, Windows builds 1909 and 1903 already have .NET Framework 4.8 installed. When you try to reinstall the software, the installation fails.

Cause

This problem is caused by a prerequisite check that was introduced in Cumulative Update 4. This process checks incorrectly for .NET Framework 4.8. Because the prerequisite check doesn’t recognize that .NET Framework 4.8 is already installed.

Status

Microsoft is researching this problem and will post more information in this article when it becomes available.

Workaround

Important

Follow the steps in this section carefully. Serious problems might occur if you modify the registry incorrectly. Before you modify it, back up the registry for restoration in case problems occur.

  • Start “regedit.exe” as an administrator.
  • Locate the following registry subkey:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full

  • Right-click the subkey, and select Permissions from the shortcut menu.
  • Select Advanced.
  • In the Advanced Security Settings window, locate the Owner attribute at the top of the window.
  • Select Change next to the listed owner.
  • In the Enter the object name to select field, enter the name of the local administrator group. For example, enter <computername>\Administrator. Then, select OK.
  • In the Advanced Security Settings window, select the local administrator group that you changed ownership to, and then select Edit.
  • Change the basic permissions to Full Control.
  • Select OK three times to save the changes and return to the main Registry Editor window.
  • Locate the following key in the path:

Name: Release
Type: REG_DWORD
Data: 528040 (decimal)
Change the Data value to 528049 (decimal)

  • Rerun the Exchange system prerequisites check, and deploy or update Exchange Server 2019.
  • Start Registry Editor, locate the subkey that’s mentioned in step 2, repeat the necessary steps to locate the Release key, and then revert the Data value to 528040 (decimal).

This should allow everything to run correctly after the installation.

POSITIVE DAY TO YOU!
FINE = Focused Intent Not Emotionalizing

REFERENCES
Exchange CU4 Prerequisite Check Fails for .NET 4.8 for Windows Server builds 1909 and 1903

Set-SendConnector cmdlet does not function correctly when updating a Send Connector on an Edge Server in an Exchange Hybrid Deployment

I have run into this issue at a number of my customers that utilize an Exchange Edge Server in their Hybrid Deployment. They’ll need to modify their send connectors for their forced TLS communication with their partners or own mailboxes in Office365. Whenever they want to modify the send connector and save the changes, they get the following error messages:

Symptoms

“PowerShell failed to invoke ‘Set-SendConnector’: Error 0x5 (Access is denied) from cli_GetCertificate”

or

“Error 0x6ba (the RPC server is unavailable) from cli_GetCertificate”

This issue occurs after you install the Cumulative Update 14 for Exchange Server 2016Cumulative Update 13 for Exchange Server 2016, or Cumulative Update 23 for Exchange Server 2013.

Cause

This issue occurs because the TLS certificate check (in case the TlsCertificateName attribute is populated on the send connector) doesn’t work against the Edge servers as the RPC communication is blocked against the Edge servers.

Workaround

Now the current workaround for this has been to delete the Edge Send Connector and recreate the connector from scratch via PowerShell with all the settings and changes entered. This is not a viable solution especially if your communications with your partners change constantly and changes are made to the secure communications channel between you and them.

Resolution

To fix this issue, install one of the following updates:

For Exchange Server 2019, install the Cumulative Update 4 for Exchange Server 2019 or a later cumulative update for Exchange Server 2019.

For Exchange Server 2016, install the Cumulative Update 15 for Exchange Server 2016 or a later cumulative update for Exchange Server 2016.

For Exchange Server 2013, there is no fix at this time. My personal recommendation is to plan an upgrade to Exchange 2019.

KEEP POSITIVLY MOVING FORWARD!

REFERENCES
Set-SendConnector doesn’t work for Exchange Server in hybrid scenarios with Edge Server installed

Exchange Server Quarterly Updates

Support Announcement:
Released: December 2019 Quarterly Exchange Updates
Release Date: December 17, 2019

Summary
Today Microsoft is announcing the availability of quarterly servicing cumulative updates for Exchange Server 2016 and 2019. These updates include fixes for customer reported issues as well as all previously released security updates.

Setup Now Requires .NET Framework 4.8
As previously announced .NET 4.8 is now required and enforced by setup with the updates released today.

Calculator Updates
Cumulative Update 4 includes a significant update to the Exchange 2019 sizing calculator. After the initial re-work and optimization for Exchange 2019 previously delivered, we’ve updated some formulas based upon new Big Funnel performance data gathered from the O365 service and real-world customer experiences. Version 10.3 of the calculator includes improvements to calculations and default settings which allow for better and smoother utilization of disk resources. We’ve received feedback from customers that they’d like more information on constraints which impact system design, specifically disk resources. Included in this update, is an indication on the Input worksheet will provide information as to whether the design is constrained by IOPs throughput or disk capacity. 

We’ve added additional explanatory messages when the calculator detects a setting conflict, made additional improvements in input performance and improved support for using manual/override configurations. The Volume Design sheet had a complete re-work to improve the presentation and accuracy of the information being displayed to support these changes. All-in-all, this version of the calculator provides the best possible experience to plan your Exchange 2019 deployment and replaces all previous releases.

Address Book Policy Changes
When organizations deploy Address Book Policies to users they can sometimes hit an issue when a locally logged in user without a mailbox tries to open a mailbox linked to another user account using Outlook. This conflict results in ABP’s being inconsistently applied. The updates released today contain a change detailed in KB4532747 which resolves this issue and ensures the ABP’s assigned to the mailbox being opened are always used.

Release Details
The KB articles that describe the fixes in each release and product downloads are available as follows:

• Exchange Server 2019 Cumulative Update 4 (KB4522149), VLSC Download
• Exchange Server 2016 Cumulative Update 15 (KB4522150), Download

Additional Information
Microsoft recommends all customers test the deployment of any update in their lab environment to determine the proper installation process for your production environment. For information on extending the schema and configuring Active Directory, please review the appropriate documentation. Also, to prevent installation issues you should ensure that the Windows PowerShell Script Execution Policy is set to “Unrestricted” on the server being upgraded or installed.
To verify the policy settings, run the Get-ExecutionPolicy cmdlet from PowerShell on the machine being upgraded. If the policies are NOT set to Unrestricted you should use the resolution steps in KB981474 to adjust the settings.
Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, or who are using Exchange Online Archiving (EOA) with their on-premises Exchange deployment are required to deploy the currently supported cumulative update for the product version in use, e.g., 2013 Cumulative Update 23; 2016 Cumulative Update 15 or 14; 2019 Cumulative Update 4 or 3.

For the latest information on Exchange Server and product announcements please see What’s New in Exchange Server and Exchange Server Release Notes.

Note: Documentation may not be fully available at the time this post is published. Article Link

KEEP POSITIVLY MOVING FORWARD!

Hyper-V: Cannot Delete a Checkpoint Due To Catastrophic Failure

I had a VM that I had restore in my environment that failed. I had to rebuild the VM and started backing up again. But since then, I have had issues with the checkpoints and kept getting these errors in my backup logs:

Catastrophic Failure to delete the checkpoint.

So. I go into Hyper-V Manager and try to manually delete the checkpoint. I got the same error:

Virtual machine failed to generate VHD tree: ‘Catastrophic failure'(‘0x8000FFFF’)

So, I go and find a blog post explaining how to manually export the checkpoint files to a new VHD and recover the VM in its current state properly so that my backups can start again. Here are the steps:

NOTE: This process will merge changes so previous checkpoints will no longer be available for rollback.

Export the last checkpoint of the VM:

Locate the most recent snapshot and select it.
Click Export from the actions menu.
Export the VM to a new location.
Shutdown the original VM.
Once the export completes you will have a new merged vhdx!

Replace the Offending VM with the Exported VM:


Click Import Virtual Machine.
The VM will have the name of the snapshot.
Power the imported VM on and validate it’s working as desired.
Power Off the VM.
Once satisfied with the new VM, delete the offending VM and it’s disks.
Rename the newly imported VM.
Place the virtual disks in their original spots and reconfigure the new VM to go to those locations.
Now you’re VM is updated and fixed!

I luckily had enough disk space on my drive to export the VM since it is my WSUS server. I probably could have just deleted the WSUS repository disk, but I did not want to chance it since the other was working. Things are back to their normal, POSITIVE state!

POSITIVE THOUGHTS AND ACTIONS STAY!
HAPPY TROUBLESHOOTING!

REFERENCES:
Hyper-V Catastrophic Failure when trying to restore a checkpoint.

Outlook Web App (OWA) HTTP to HTTPS Redirection

For most companies today, we want to make access to OWA easy for the users. Most folks will just type in mail.domain.com/owa or something of the like to get to the OWA page. If you don’t use HTTPS by default though, you will not be able to access OWA and will get an error on the page. We need to be able to redirect the HTTP query to go to SSL or HTTPS so that you get the proper logon page and have the access secured by SSL PKI as per the security standard.
Now, most bigger companies will install a load balancer that will program the redirection to HTTPS when the request is made before it hits the Exchange Server. But, for small companies, like mine, that cannot afford a load balancer, we need a native way in Windows and Exchange to be able to perform the same task and have it redirect to HTTPS so that your users are not confused when typing in the address.

The following shows how to configure IIS so that it natively redirects all HTTP requests for OWA to HTTPS.

By default in Exchange Server, the URL https://<ServerName> redirects users to https://<ServerName>/owa. But, if anyone tries to access Outlook on the web (formerly known as Outlook Web App) by using http://<ServerName> or http://<ServerName>/owa, they’ll get an error.

You can configure http redirection for Outlook on the web so that requests for http://<ServerName> or http://<ServerName>/owa are automatically redirected to https://<ServerName>/owa. This requires the following configuration steps in Internet Information Services (IIS):

  1. Remove the Require SSL setting from the default website.
  2. Restore the Require SSL setting on other virtual directories in the default website that had it enabled by default (except for /owa).
  3. Configure the default website to redirect http requests to the /owa virtual directory.
  4. Remove http redirection from all virtual directories in the default website (including /owa).
  5. Reset IIS for the changes to take effect.

Step 1: Use IIS Manager to remove the Require SSL setting from the default website

  1. Open IIS Manager on the Exchange server. An easy way to do this in Windows Server 2012 or later is to press Windows key + Q, type inetmgr, and select Internet Information Services (IIS) Manager in the results.
  2. Expand the server, and expand Sites.
  3. Select Default Web Site. and verify Features View is selected at the bottom of the page.
  4. In the IIS section, double-click SSL Settings.
    SSL1
  5. On the SSL Settings page, clear the Require SSL check box, and in the Actions pane, click Apply.
    SSL2

Note: To perform this procedure on the command line, open an elevated command prompt on the Exchange server (a Command Prompt window you open by selecting Run as administrator) and run the following command:

Step 2: Use IIS Manager to restore the Require SSL setting on other virtual directories in the default website

When you change the Require SSL setting on a website in IIS, the setting is automatically inherited by all virtual directories in the website. Because we’re only interested in configuring Outlook on the web, you need to restore the Require SSL setting for other virtual directories that had it enabled by default.

Based on the information in the Default Require SSL and HTTP Redirect settings in the default website on an Exchange server section, use the following procedure to restore the setting on the other virtual directories where Require SSL was enabled by default:

  1. In IIS Manager, expand the server, expand Sites, and expand Default Web Site.
  2. Select the virtual directory, and verify Features View is selected at the bottom of the page.
  3. In the IIS section, double-click SSL Settings.
    SSL3
  4. On the SSL Settings page, select the Require SSL check box, and in the Actions pane, click Apply.
    SSL4
  5. Repeat the previous steps on each virtual directory in the default website that had Require SSL enabled by default ***(except for /owa)***. The only virtual directories that don’t have Require SSL enabled by default are /PowerShell and /Rpc.

NOTE: PLEASE REMEMBER TO NOT CHECK THE “Require SSL” FOR THE /OWA DIRECTORY. THIS WILL CAUSE A 403 Access Denied ERROR WHEN TRYING TO REDIRECT.

Note: To perform these procedures on the command line, replace <VirtualDirectory> with the name of the virtual directory, and run the following command in an elevated command prompt:

Step 3: Use IIS Manager to configure the default website to redirect to the /owa virtual directory.

  1. In IIS Manager, expand the server, and expand Sites.
  2. Select Default Web Site. and verify Features View is selected at the bottom of the page.
  3. In the IIS section, double-click HTTP Redirect.

  4. On the HTTP Redirect page, configure the following settings:
  5. Select the Redirect requests to this destination check box, and enter the value /owa.
  6. In the Redirect Behavior section, select the Only redirect requests to content in this directory (not subdirectories) check box.
  7. In the Status code list, verify Found (302) is selected.When you’re finished, click Apply in the Actions pane.

Note: To perform this procedure on the command line, open an elevated command prompt and run the following command:

Step 4: Use IIS Manager to remove http redirection from all virtual directories in the default website

When you enable redirection on a website in IIS, the setting is automatically inherited by all virtual directories in the website. Because we’re only interested in configuring redirection for the default website, you need to remove the redirect setting from all virtual directories. By default, no directories or virtual directories in the default website are enabled for redirection. For more information, see the Default Require SSL and HTTP Redirect settings in the default website on an Exchange server section.

Use the following procedure to remove the redirect setting from all virtual directories in the default website (including /owa):

  1. In IIS Manager, expand the server, expand Sites, and expand Default Web Site.
  2. Select the virtual directory, and verify Features View is selected at the bottom of the page.
  3. In the IIS section, double-click HTTP Redirect.

  4. On the HTTP Redirect page, change the following settings:
  5. Clear the Only redirect requests to content in this directory (not subdirectories) check box.
  6. Clear the Redirect requests to this destination check box.
  7. In the Actions pane, click Apply.

  8. Repeat the previous steps on each virtual directory in the default website.

Note: To perform these procedures on the command line, replace <VirtualDirectory> with the name of the virtual directory, and run the following command in an elevated command prompt:

Step 5: Use IIS Manager to restart IIS

  1. In IIS Manager, select the server.
  2. In the Actions pane, click Restart.

Note: You can also perform an IISRESET from and Elevated PowerShell Prompt.

My biggest take away from this was NOT setting the SSL Requirement Properly in the /owa directory when configuring this. By default, the setting is to Require SSL, but to redirect properly, you have to have that Virtual Directory in IIS set to NOT Require SSL. Having the 403 error was driving me crazy. I had to get someone else to look at it, but they didn’t catch it either! That is why I made a point to write this article with the /owa catch in mind. I hope this helps!

HAPPY CONFIGURATION!
POSITIVE LIFE WILL BRING SUCCESS!

REFERENCES:
Configure http to https redirection for Outlook on the web in Exchange Server
Default Require SSL and HTTP Redirect settings in the default website on an Exchange server

Exchange 2010 Extended Support will end on October 13th, 2020

I wanted to pass this announcement along to everyone so that they are aware of the support ending for Exchange 2010. I personally have noticed a large number of Exchange 2010 environments starting to show age as the newer Outlook clients are having performance issues with Exchange 2010. If your team has not planned an upgrade to Exchange 2016 (you cannot upgrade directly from Exchange 2010 to 2019), I would advise that your team do so very soon. Exchange 2010 has been a great product for many years, but it is finally time for it to retire and allow the next generation of Messaging Services take the stage.

Formal Announcement:

Exchange 2010 End of Support extended to October 2020

Announced today, and in alignment with Office 2010 and SharePoint 2010, and after investigating and analyzing the deployment state of an extensive number of Exchange customers, Microsoft has decided to move Extended Support date for Exchange Server 2010 from January 14th 2020 to October 13th 2020.
After October 13th 2020, Microsoft will no longer provide technical support for problems that may occur with Exchange 2010 including:

– bug fixes for issues that are discovered and that may impact the stability and usability of the server
– security fixes for vulnerabilities that are discovered and that may make the server vulnerable to security breaches
– and time zone updates

Customer installations of Exchange 2010 will, of course, continue to run after this date; however, due to the changes and potential end of support risks, Microsoft strongly recommends customers migrate from Exchange 2010 as soon as possible.

FAQ’s

  • Can customers upgrade directly to Exchange 2019?
    Customers cannot upgrade directly from Exchange 2010 on-premises to Exchange Server 2019. They may upgrade to Exchange 2013 or 2016 directly from Exchange 2010 and we of course recommend Exchange 2016.
  • Since Exchange 2010 runs on Server 2008 and 2008R2, are those operating systems still supported?
    On January 14, 2020, support for Windows Server 2008 and 2008 R2 will end. That means the end of regular security updates for these Windows customers. Since Exchange Server 2010 runs on top of Windows Server 2008 and Windows Server 2008 R2, it’s important for customers to consider how they will obtain security updates for the underlying operating system. Extended Security Updates for Server 2008 and 2008 R2 are now available for purchase and can be ordered from Microsoft or a Microsoft licensing partner. The delivery of Extended Security Updates (ESU) will begin after the End of Support dates, if and when available. 
  • Does Microsoft support Exchange 2010 on any other Server versions?
    Exchange Server 2010 SP3, with Update Rollup 26 or higher, installed on Windows Server 2012 R2 is supported until October 13, 2020.
  • That didn’t quite answer my question. If a customer calls between January 14 and October 13 2020, and is running Exchange 2010 on Server 2008 or 2008 R2, and does not have an ESU for Windows, can they still be assisted?
    Yes. Per the Lifecycle FAQ.
    If I am running a Microsoft product that is currently supported under the Lifecycle Policy, but my operating system is no longer supported, can I still receive support?
    If the problem is specific to the Microsoft product and it is within the Lifecycle Policy, Microsoft will provide support.
    If the problem is a result of the combination of the operating system and the Microsoft product, the problem will not be supported.
    More simply::
    Exchange 2010 on Server 2008 or 2008 R2: Starting January 14, 2020, provide support until a proven issue is found with the OS. This ends in October 2020.
    Exchange 2010 SP3 RU26+ on Server 2012 R2: We support regardless, but Exchange support still ends in October 2020.

  • Will Microsoft be offering Extended Support Updates (ESU’s) for purchase for Exchange 2010 customers?
    No.
  • What resources are available for customers? 
    – An upcoming Exchange Team blog post, titled “Exchange On-Premises Best Practices for Migrations from 2010 to 2016,” will provide great technical guidance for customers and support agents with their on-premises migrations.  
    – If migrating to Office 365 and Exchange Online, customers may be eligible to use the free Microsoft FastTrack service. FastTrack provides best practices, tools, and resources to make migration to Office 365 and Exchange Online as seamless as possible.
    – For customers that run into any problems during their migration to Office 365 and are not eligible for FastTrack, or if migrating to a newer version of Exchange Server, customers can of course utilize Support or the Exchange Technical Community.
    – Customers may also choose to engage a partner to help.  Microsoft has a great number of partners with deep skills in Exchange, and you can browse a list of Exchange partners at
    https://www.microsoft.com/en-us/solution-providers/home.

HAPPY UPGRADING!
CONTACT ME FOR QUESTIONS CONCERNING UPGRADING YOUR EXCHANGE ENVIRONMENT!

Create a custom Windows 10 image for distribution using and ISO image.

I’ve been currently assisting with onboarding at my new Full Time Contractor position at Microsoft. All of the new FTCs received laptops and needed to have the newest build of Windows 10 installed. The issue was that all of our laptops came with Windows 10 Professional and we needed to upgrade them to Enterprise edition.

After finding a working key for Enterprise Edition, we were still having issues joining the MS Azure domain so that we could get all of the needed software properly to being onboarding with Microsoft.

So, after going through a couple of re-images of my laptop with some failures attached to that, I finally was able to get the process down so that time would not be waisted for the oncoming new hires once they received their laptop. The issue was getting the correct build of Windows 10 and getting the proper apps installed in an efficient manner. Since the onboarding process was quickly moving, I needed to find a way to help streamline the process so the others would not have to go through all the mess I went through to get everything setup.

So, I began looking for a way to create a customized ISO for the build that would already have apps, settings, and customizations installed. I found this great article that details the process. I wanted to re-post this article here showing the steps I took to create the customized image by creating a VM in Hyper-V and then converting that completed image to an ISO that could be downloaded and utilized for the installation.

Creating a customized ISO image with pre-installed software and no user accounts

  • A generalized ISO image without any pre-set user accounts, with pre-installed software, desktop, File Explorer and Start customizations will be created.
  • All customizations and personalization will automatically be applied to all new user accounts
  • Clean install will perform a normal OOBE, asking for regional settings, initial user and so on
  • This ISO will be generalized meaning it is hardware independent and can be used to install Windows on any computer capable of running Windows 10, regardless if the machine is a legacy BIOS machine with MBR partitioning, or a UEFI machine with GPT partitioning
  • The ISO image will be bootable on both BIOS / MBR and UEFI / GPT systems

NOTE: This post will show how to use a virtual machine to create the ISO. All virtual machine references and instructions in this tutorial apply to Hyper-V, available in Windows 10 PRO, Education and Enterprise editions. Oracle VirtualBox and VMware users might need to consult their preferred virtualization platform’s documentation if instructions can’t be used as is.
Everything in this instruction can be made in each edition of Windows 10 (in Home and Single Language editions using a third party virtualization platform) with native Windows tools and programs, apart from Windows Deployment and Imaging Tools, part of Windows 10 Assessment and Deployment Kit (ADK) needed later in the post. The ADK is a free native Microsoft tool, downloadable directly from Microsoft.
If you will do this on a Hyper-V virtual machine (which is the recommended method), make sure to set the new virtual machine to use Standard Checkpoints instead of default Production Checkpoints. You can do this in virtual machine’s settings:

Name:  image.png
Views: 109048
Size:  81.9 KB
Use Standard Checkpoints
Virtual machine generation is irrelevant, you can use Generation 1 or 2 as you wish

This method will produce an ISO image which can be compared to any original Windows 10 ISO you download from Microsoft, apart from the fact that it already contains pre-installed software according to your choice. It will also contain a customized and personalized default user profile, the base Windows uses whenever a new user profile will be created.

A customized default user profile means that whenever a new user account is created, all customizations (Start tiles, File Explorer & desktop icon and view settings, colors, wallpaper, theme, screensaver and so on will be applied to new user profile instead of Windows defaults.

Installation using this ISO will take somewhat longer than using a standard ISO because it not only contains full Windows setup, but also the pre-installed software. Notice that depending on how much space pre-installed software takes, you might not be able to burn this ISO to a standard 4.7 GB DVD disk but have to use a dual layer disk or a USB flash drive instead. My customized image came out to be about 8.5 GB in size.

The ISO created will include no user profile folders, personal user data and files.

This ISO image can be used on any hardware setup capable of running Windows and can be shared, subject to people you share the ISO with have valid licenses and / or activation keys for both Windows 10 and pre-installed software.

System Preparation Procedure

  • Download the Windows ISO Installation tool from Microsoft
    • Use this TOOL to download the ISO and create the installation media
  • Install Windows 10 on your VM using the downloaded ISO

NOTE: The normal Windows Download from the link above will download Windows 10 Professional. You will need a key for the installation to upgrade to Enterprise Edition and you will need to be able to activate the copy of Windows to be able to save the customizations you create for your ISO.

  • Boot into Windows 10 and do the following:
    • Activate the Windows Edition your are installing with your key. You will require internet connectivity. I needed Enterprise Edition so I changed the Product Key In Settings to upgrade it from Professional.
    • Install your preferred software, customize and personalize Windows, remove / add Start tiles as you wish, and set your preferred group policies (group policies not available in Home and Single Language editions). Do not run any program you install!
    • Update all software and run Windows Update to get all the latest updates for the image.
    • Notice that software installed now will be included in ISO install media, and will be pre-installed for all users on each computer you install Windows to using this custom ISO.

NOTE: If Windows on your reference machine is not activated, you cannot personalize it. In this case you need to modify Windows theme (wallpaper, screensaver, colors, sounds) as you wish on another, activated Windows 10 machine, save the theme as a theme file, copy it to inactivated reference machine and apply (double click).
Also notice that Edge as well as other UWP apps do not work when signed in to built-in admin account. If you need a browser to download software you have to use a third party browser or Internet Explorer. IE can be started from Run dialog by typing iexplore and clicking OK.

  • Open an elevated command prompt and enter the following:

Windows will now restart in Audit Mode using built-in administrator account. You will see a Sysprep prompt in the middle of display:

Name:  image.png
Views: 107212
Size:  73.1 KB
Sysprep Program Window
Leave it open for now
  • Open Notepad, paste the following code to it, make the necessary changes to customize the installation, and save it as
    File name: unattend.xml (exactly this name!)
    Save as type: All files (important!)
    Save in folder: C:\Windows\System32\Sysprep

  • When Sysprepping with the Generalize switch, which we will soon do, the component CopyProfile being set to be TRUE in answer file has a small issue or rather a small inconvenience: it leaves the last used user folders and recent files of built-in admin to end user’s Quick Access in File Explorer.
  • To fix this, we need to reset Quick Access to default whenever a new user signs in first time. It requires the need to run a small batch file at first logon of new user, and then remove the batch file itself from user’s %appdata% so Quick Access will not be reset on any subsequent logon.
  • To do this, open an elevated (Run as administrator) Notepad (Notepad must be elevated to save in system folders), paste the following code to it, save it as:
    File name: RunOnce.bat (or any name you prefer, with extension .bat)
    Save as type: All files (important!)
    Save File in folder: %appdata%\Microsoft\Windows\Start Menu\Programs\Startup

  • Delete all existing user accounts and their user profile data (Option One in this tutorial)
  • You are currently signed in using Windows built-in administrator account. In File Explorer, open C:\Users\Administrator folder and check that all user folders are empty deleting all possibly found content
  • Run Disk Clean-up, selecting and removing everything possible (tutorial)
  • When the disk has been cleaned, create a checkpoint of the VM from Hyper-V Manager. Right Click VM > Click Checkpoint
Manual Checkpoint from Hyper-V Manager
  • In Sysprep dialog still open on your desktop, select System Cleanup Action: Enter System Out-of-Box Experience (OOBE), select Shutdown Options: Shutdown, select (tick the box) Generalize, click OK:
Name:  image.png
Views: 110675
Size:  14.0 KB
Sysprep Selected Options Before Shutdown

Sysprep will now prepare Windows, shutting down machine when done. LEAVE THE VM OFF AND DO NOT RESTART IT! Now, we continue to the Image Creation section.

Name:  image.png
Views: 110209
Size:  3.4 KB
Sysprep Preparing the Machine

Image Creation Procedure

  • On your Hyper-V Host machine, open Disk Management
  • Select Attach VHD from Action menu:
Name:  image.png
Views: 109944
Size:  18.3 KB
  • Browse to and select your reference virtual machine’s VHD / VHDX file. If you have any checkpoints (AVHD / AVHDX files) created on this vm, select the one with most recent time stamp. Note that you have to select show all files to be able to see checkpoint AVHD / AVHDX files:
Click image for larger version.   Name:	image.png 
Views:	987 
Size:	94.6 KB 
ID:	113169
Select the most recent time stamped file
  • Select the check box labeled Read-only (this is very important!), then click OK:
Name:  image.png
Views: 110148
Size:  15.3 KB
BE SURE TO SELECT READ-ONLY

IMPORTANT: Forgetting to select Read-only will especially when mounting a checkpoint AVHD / AVHDX file make it unusable for Hyper-V! You will NOT be able to boot your VM and could corrupt it should you have write access on the mounted VHDX file.
Windows mounts the virtual hard disk, and all of its partitions, as separate disk. In case of an MBR disk it even mounts the system reserved partition.

Name:  image.png
Views: 110574
Size:  6.3 KB

NOTE: In the above picture the Windows system partition for the reference VM is drive K:

  • Open the Windows system partition VHD to be sure that’s the one where Windows is installed, note the drive letter your host assigned to it.
  • Open an elevated Command Prompt, enter the following command to create a new install.wim file:

NOTE: D:\install.wim path in this case is the drive and directory where you want to save the image file. K:\ path is the capture path with subfolders of the drive you want to image FROM

Click image for larger version.   Name:	image.png 
Views:	907 
Size:	26.5 KB 
ID:	113172
dism command

NOTE: The name given in /name switch in above command is irrelevant as we will name the ISO later on, but is needed for the command to run. Use any name you want to for the switch parameter.
The image process will take time, go get something to eat as I did. On my high end Hyper-V server this takes over 20 minutes, the first half of it without showing any progress indicator whatsoever. DISM works somewhat faster if you don’t use optional switches /checkintegrity and /verify but it is not recommended that you to create install.wim without checking its integrity and verifying it.

  • When completed capturing the image, detach the VHD / VHDX or AVHD / AVHDX file from host by right clicking it in Disk Management and selecting Detach VHD:
Name:  image.png
Views: 109850
Size:  28.5 KB
Detach the VHDX

ISO Image Creation Procedure

  • Mount the original Windows 10 ISO you downloaded in the first step to a Virtuial Drive on your Hyper-V Server Host.
  • Copy its contents (everything) to a folder you create on one of your Hyper-V host’s hard disks:
Click image for larger version.   Name:	image.png 
Views:	1221 
Size:	463.8 KB 
ID:	125114

I named the folder ISO_Files and placed it on the D: drive where I had created the image from the previous section. Alternatively, you can copy the contents of a created Windows 10 install USB or DVD to the ISO_Files​ folder.

  • Browse to your custom install.wim created earlier in previous section. Copy it to Sources folder under the ISO_Files folder, replacing the original install.wim in that directory:
Name:  image.png
Views: 110062
Size:  55.8 KB
Note

IMPORTANT: If the ISO you used when copying the files to the ISO_Files folder has been made with Windows Media Creation Tool, the ISO_Files\Sources folder contains an install.esd file instead of install.wim.
In this case you will naturally not get “File exists” prompt. Simply delete the install.esd file and paste your custom install.wim to replace it.
This will help reduce the overall size of the ISO and not confuse the installation process when ran.

  • Now, we download the latest Windows Assessment and Deployment Kit(ADK)for Windows 10: Windows ADK downloads – Windows Hardware Dev Center
    The full download for the ADK is about 7.5 GB but luckily we only need the Deployment Tools portion. So, unselect everything else except Deployment Tools and click Install:
Name:  image.png
Views: 106320
Size:  348.3 KB
Select Only Deployment Tools for the Installation
  • Once completed, you should have a folder within your start menu for the ADK Tools Installation under the folder Windows Kits. Start the Deployment and Imaging Tools interface program by Running the Program as an Administrator:
Name:  image.png
Views: 110029
Size:  38.8 KB
Right Click the Application and “Run As Administrator”
  • At the command prompt, type cd\ to bring your prompt to the root of the folder path you are on.
  • Type the following command to initiate creation of the ISO image file:

Replace three instances of d:\iso_files with the path to the ISO_Files folder where you copied Windows installation files. Notice that this is not a typo: first two of these instances are typed as argument for switch -b without a space in between the switch and argument. This is to tell the oscdimg command where to find boot files to be added to ISO.

Replace d:\14986PROx64.iso with the path where you want to store the ISO image. This is where you also name the ISO file what you want the file name to be.

Although the command seems a bit complicated, everything in it is needed. For more information about the oscdimg command line options, see: Oscdimg Command-Line Options

Click image for larger version.   Name:	image.png 
Views:	1467 
Size:	54.6 KB 
ID:	113179
Screenshot of the OSCDIMG command ran

You now have a completed ISO image ready for distribution to your machines. The overall process took me about 4 hours to complete with all the customizations that I did. Thanks again to Ten Forums for the article. I have provided references below for your convenience as well.

HAPPY IMAGING!!
PLEASE COMMENT!!

REFERNCES:
Create Windows 10 ISO image from Existing Installation
Open and Use Disk Cleanup in Windows 10
Download Windows 10
Windows 10 sysprep – how to skip entering product key
Windows ADK downloads – Windows Hardware Dev Center

The Windows Time Service, Hyper-V Hosts, and DCs that are VMs.

The sheer craziness of it all! I noticed that my clocks were off on my servers by FOUR minutes. I had originally set in group policy for the PDC emulator for my domain, a VM on one of my Hyper-V hosts, to get the time from the Public NTP hosts. I then configured a group policy to have all the other machines get their time from the PDC Emulator.

This was working great for me until I realized that my Hyper-V hosts were actually controlling the time of the VMs. They were also configured to get the time from the PDC Emulator, but essentially, due to how Hyper-V is configured, the PDC Emulator VM was getting the time from the Host. So, once the time got thrown off, everything went wacky on me!

I’d read through a couple of articles and found the configuration flaw of Hyper-V and the need for those servers to get their time from the external NTP hosts as well as be configured as NTP servers themselves. This totally went against my Group Policy configuration which caused the issue!

Luckily, I had a stand alone server that is a tertiary DC in the domain not running Hyper-V. I was able to get my time synced again properly after performing the following configuration.

  • I had to move the FSMO roles to the tertiary DC with the following cmdlet:

  • I then made sure the tertiary DC was syncing time correctly by running the following on that server:

  • I then removed the Group Policy Object for syncing the time source to the DC that I had linked to my Hyper-V Servers OU in Active Directory
  • Ran a gpupdate /force on the Hyper-V host to remove the policy there
  • I then had to reconfigure the Hyper-V hosts to be NTP Servers and clients that got their time from a public NTP server:

The one problem Hyper-V host that was syncing with the DC VM would not change settings via Group Policy nor through the w32tm cmdlet. I even went into the registry and tried to modify the following keys to make the changes stick:

The values would just not change, most likely due to the time not being synchronized. I had to reboot the server and then run through the process again in order for the changes to stick.

I did look at another article that said to do the following on the DC VM in order for time NOT to sync with the Hyper-V Host:

Go into Hyper-V console on the host machine, right-click on the client VM AD server, and select Settings. Once in here, on the left look under:

Management –> Integration Services
Untick Time Synchronization
Click Apply/OK

Virtual Machine Settings within Hyper-V

Things are running smoothly now. Please view the references at the bottom of the post. There are a couple of great articles about the Time Synchronization process with Hyper-V and why it needs to be setup the way I have it now. I wished I had read it before I originally set this up. I will post the article about getting group policy to handle the time sync process. Just remember, if your PDC Emulator is a VM, don’t sync it to a public NTP server. Sync it to your Hyper-V Host and have the Host sync publicly.
In the long run, I think it is a good design solution to have your Hyper-V hosts time synced to the Public NTP servers than having to remember to configure each VM DC you create to NOT time sync with the host. To each is own though, and one thing I learned from working Microsoft, there are multiple ways to get to the same goal that are technically sound methods.

THANKS FOR READING!
PLEASE COMMENT!

REFERENCES:
Setup of NTP on Hyper-V servers
Time Synchronization in Hyper-V
“It’s Simple!” – Time Configuration in Active Directory
NTP Circular Time Sync – Windows Server 2012 R2 / Hyper-V

Exchange Server Client Access URL Configuration Script

In my career, I have to be able to be efficient as most of my projects are on a time crunch schedule. Being able to quickly configure Exchange when setting up a server environment is crucial to the success of the project.

While still honing my skills in PowerShell, I was attempting to create my own script to help configure all of the Virtual Directories in one shot rather than go to each setting and configure them manually. It did not go very well, so as I do, I research and find great professionals that do great work in scripting so that I may learn from them.

In doing so, I found Paul Cunningham’s script that performs this. I took the following script and modified it to add the PowerShell Virtual Directory to it as I like to configure that as well.

***YOU CAN REM THE LINES OUT SHOULD YOU NOT WANT TO CONFIGURE THAT DIRECTORY***

Here is my version of the script:

NOTES:

  • PowerShell script to configure the Client Access server URLs for Microsoft Exchange Server 2013/2016. All Client Access server URLs will be set to the same namespace.
  • If you are using separate namespaces for each CAS service this script will not handle that.
  • The script sets Outlook Anywhere to use NTLM with SSL required by default.
  • If you have different auth requirements for Outlook Anywhere use the optional parameters to set those.
  • The script sets PowerShell to use Basic with SSL required by default.
  • If you have different authentication requirements for PowerShell use the optional parameters to set those.
  • PowerShell was added to the settings. Please be sure to REM those lines of code should you NOT want to configure the PowerShell Virtual Directory.

USAGE:

HAPPY SCRIPTING!
POSITIVE ENERGY!
PLEASE COMMENT!

REFERENCES:
Exchange Server Client Access URL Configuration Script
PowerShell Script to Configure Exchange Server Client Access URLs

Installing an ‘IP-less’ Exchange Server 2019 Database Availability Group

Yesterday, I posted on how Exchange now uses the Resilient File System (ReFS) to optimize and protect Exchange critical files. Another layer of protection is using a database availability group (DAG) for redundancy and is a necessary factor when designing an Exchange Enterprise Environment.
In this example, I will walk you through the installation of an Exchange Server 2019 DAG as I configured in my environment. This DAG will contain two Exchange Servers in the same site with a third Windows Server 2019 server being the File Share Witness (FSW).

Two Server Exchange DAG Configuration

For my configuration, I configured two identical Windows Server 2019 VMs (same procs, RAM, vhdx drives, partitions, etc…). I configured the Exchange Data Volume using ReFS and mounted them to the same folder on the C: Drive on each server. This is very important for replication to take place successfully when the databases are added to the DAG.


I next went to the Admin server where the FSW would be hosted and added the Exchange Trusted Subsystem Account to the local Administrators group on that server:

IMPORTANT!
Add the Exchange Trusted Subsystem Account to the Local Administrators Group on the FSW.

NOTE: The reason that this is an ‘IP-less’ DAG is that I’m creating a DAG with no cluster administrative access point (CAAP). The DAG has no IP address of its own, and no computer object in Active Directory. The main implication of this is that backup software that relies on the CAAP or backup operations won’t work. This option of an ‘IP-less’ DAG was first introduced in Exchange Server 2013 SP1/CU4, so by now any decent backup products should support this configuration. But you should always verify this with your backup vendor of choice. Also be aware that this is only supported for DAGs that are running on Windows Server 2012 R2 (or later).

Next, we create the DAG from Exchange PowerShell using the New-DatabaseAvailabilityGroup cmdlet. Now remember that since you are using the ReFS system for your database volumes, you will need to specify the -FileSystem parameter within the cmdlet to assure proper setup and replication of the data files.

Next, we add the Exchange Servers that hold the databases that will be replicated within the DAG:

The DAG will now show the two servers as Operational Member Servers:

The FSW Directory was created on the admin01 server when the DAG was created. We can verify that with the following cmdlet:

Next, we add the databases that we want replicated to the DAG as replicated databases. I want all my Databases on EX01 to replicate to EX02 and vice versa for the EX02 Databases. I want the activation preference to remain on the server that the databases were originally created on so I will use the -ActivationPreference parameter to accomplish that. I will go into more detail on Activation Preference in another post.

Now we verify that the Database Copies are healthy on each replication member using the Get-MailboxDatabaseCopyStatus cmdlet. You will see a Healthy Status on the replicated copies:

POSITIVE ENERGY!
KILL NARCISSISM!
HAPPY TROUBLESHOOTING!

REFERENCES:
Installing an Exchange Server 2016 Database Availability Group

Using the Resilient File System for Exchange Server

In my ongoing effort for becoming more knowledgeable on Exchange Server, I found that the preferred new file system for Exchange Databases and Log files is the ReFS.
ReFS is not that new. Microsoft’s Resilient File System (ReFS) was introduced with Windows Server 2012. ReFS is not a direct replacement for NTFS, and is missing some underlying NTFS features, but is designed to be (as the name suggests) a more resilient file system for extremely large amounts of data.

Support for ReFS with Exchange Server

From Exchange Server 2013 and upwards (which includes Exchange Server 2019 today) Microsoft supports the use of ReFS for Exchange servers, and in fact they now recommend it as the preferred file system for Exchange Server 2019, within the following guidelines.

For Exchange Server 2013:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features must be disabled for the database (.edb) files or the entire volume that hosts database files.
  • Hotfix KB2853418 must be installed.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2013 installation volume, but you can consider using ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2016:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2016 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2019:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.

This means that you should continue to use NTFS for your operating system and Exchange Server 2019 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

Creating an ReFS Formatted Volume

In Windows Server during the New Volume Wizard when you get to the step for configuring File System Settings change the file system from NTFS to ReFS.

exchange-server-refs

NOTE: Using the New Volume Wizard does not give you the option to disable data integrity at the volume level. To set it at the volume level itself use PowerShell when configuring new volumes. I found this out the hard way and am now re-configuring my volumes to disable the Integrity Streams.

I needed to create the mount point to mount the volume to:

I then got a list of my available disks:

In my case, disk 2 was the one I needed to format and change. I had to create a new partition and then format it:

Once formatted, I mount the volume to the Directory created earlier:

NOTE: Partition 1 on a disk is always reserved for system files on the drive volume. So the active partitions will always start at 2.

Lastly, verify that the partition is online and that the Integrity Streams are turned off:

Additional Considerations

When you are deploying an Exchange 2016 or 2019 DAG and using Autoreseed, the disk reclaimer needs to know which file system to use when formatting spare disks. So when, creating a DAG in Exchange PowerShell, make sure to set the -FileSystem parameter. For Exchange Server 2013 DAGs, manually format the spare volumes with ReFS.

More coming soon. I will post how I setup the “IP-less” DAG for my environment and got replication functional for my Exchange Databases.

REFERENCES:
Exchange 2013 storage configuration options
Exchange 2016 Preferred Architecture
Exchange Storage for Insiders: It’s ESE (Ignite video)
ReFS Exchange Server Volumes
Preparing ReFS Volumes for Exchange

Hyper-V General Access Denied error when trying to load a Virtual Hard Drive and start a VM

I was working on setting up a VM for my server farm and mis-configured one of the vhdx drives. I ended up having to delete that drive and recreate it in Hyper-V manager. When I did though, I received an error stating that I could not start the virtual machine:

An error occurred while attempting to start the selected virtual machine(s).
‘VMName’ failed to start. (Virtual machine ID ‘SomeID’)
‘VMName’ Microsoft Emulated IDE Controller (Instance ID ‘SomeID’): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)
‘VMName’: IDE/ATAPI Account does not have sufficient privilege to open attachment ‘C:\Users\Public\Documents\Hyper-V\Virtual hard disks\DiskName.vhdx’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)
‘VMName’:  Account does not have sufficient privilege to open attachment ‘V:\Hyper-V\Virtual hard disks\DiskName.vhdx’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)

Causes

Each virtual machine is started using a virtual machine account. The virtual machine account needs read and write access to the .vhd/.vhdx file, but if the file has just been copied from somewhere then it most likely lacks the necessary file permissions.
That happened in my case because I had just created the vhdx drive and did not create it from the VM itself. I just attached it to the VM. So, when I booted the VM, it gave the error.

Remediation

There are a few ways that you could remediate the issue. The simplest way, if it is a new VM, is to remove the drive in the VM settings and then re-create it from scratch. That is what fixed it for me.
Another way is to add the VM GUID to the permissions so that it can access the vhdx file properly:

  • If you don’t already have the Hyper-V Manager error dialog open (“An error occurred while attempting to start the selected virtual machine(s) …”) then try to start the virtual machine now. You need the error open.
  • Click “See details”. This will show additional details, and will look something like:

‘PC-Name’ failed to start. (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD) ‘PC-Name’ Microsoft Emulated IDE Controller (Instance ID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
‘PC-Name’: IDE/ATAPI Account does not have sufficient privilege to open attachment ‘E:\Hyper-V\PC-Name\Virtual Hard Disks\MyVHD.vhdx’.
Error: ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
‘PC-Name’: Hyper-V Virtual Machine Management service Account does not have sufficient privolege to open attachment ‘E:\Hyper-V\PC-Name\Virtual Hard Disks\MyVHD.vhdx’.
Error: ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
Where PC-Name will be the name of your virtual PC. The long sequence of letters and numbers (in my case above “B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD”) is the Virtual Machine ID. This number is significant and you need it to fix the problem.

  • On the host server open an elevated command prompt.
  • Enter the following:

You will need to substitute the path to the vhd/vhdx file – you can obtain this from the original error message, and the Virtual-Machine-ID that you obtained from the “See details” part of the error.

So the line for me was:

NOTE: If you get the message “Failed processing 1 files” then check the virtual machine ID.

  • Now try to start the virtual machine. The error should no longer be present.

There is also a PowerShell Gallery script that is supposed to remediate this issue:

http://www.ntsystems.it/page/PS-Restore-VMPermissionps1.aspx

I haven’t tried it but it looks as it would work. Please review and leave a comment should you have issues with the script.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!
POSITIVE ENERGY!

REFERENCES:
Resolved: Hyper-V General access denied error when trying to load a Virtual Hard Drive
Restore-VMPermission
Virtual machine fails to start with General access denied error / Account does not have sufficient privilege to open attachment