Remote Desktop Licensing Mode is Not Configured when configuring Remote Desktop Services

I recently setup a backend RDP server so that I could test our remote services. I went through the installation process and installed my RDP license on the server successfully. The problem was that I was not getting an error when logging onto the server locally to check the status of the RDP Server:

remote desktop licensing mode is not configured
Error Example for RDP Licensing

I also had Events within Event Viewer:

Log Name: System
Source: Microsoft-Windows-TerminalServices-Licensing
Date: 6/24/2020 3:44:16 PM
Event ID: 18
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: SRV2016-02.ldlnet.net
Description:
The Remote Desktop license server “SRV2016-02” has not been activated and therefore will only issue temporary licenses. To issue permanent licenses, the Remote Desktop license server must be activated.

Usually, this error appears as a notification popup in the bottom right-hand corner of the screen saying:

Remote Desktop licensing mode is not configured
Remote Desktop Services will stop working in xx days. On the RD Connection Broker server, use Server Manager to specify the Remote Desktop licensing mode and the license server.

And when you click on this notification popup, it doesn’t redirect you anywhere and it gets simply disappeared which is a quite frustrating situation.

I thought I had this set properly, but the RDP Licensing Diagnoser application told me that I needed to choose the licensing configuration to distribute for Per Device OR Per User. The licenses I installed were per device so I did some research.

I found out that this had to be configured via the Registry, PowerShell, or could be configured through GPO. I chose to do GPO since that would always apply on my servers in my domain.


How To configure the Remote Desktop Licensing Mode through the Registry

Here’s how to change the licensing mode for Remote Desktop session host using the registry editor and get rid of the error message Remote Desktop Services will stop working in xx days:

At first, press Windows + R keys together and then type regedit in the Run dialog box and press Enter key.

regedit windows 10

Next, in the left pane of the Registry Editor, navigate to the following registry key:

licensing mode for the remote desktop session host is not configured

Next in the right pane, double-click on the LicensingMode to edit its value and then change the Value data according to your requirement:

Set the Value data 2 for Per Device RDS licensing mode
Set the Value data 4 for Per User RDS licensing mode

remote desktop services will stop working

Finally, click on the OK button to save the changes.

Now, restart your computer and check if the Remote Desktop licensing mode is not configured issue on Windows Server has been resolved or not.

Once you changed the licensing mode, now everything will be reported accurately and the Remote Desktop session host will recognize the licensing configuration.


How To configure the Remote Desktop Licensing Mode through a Group Policy Object (GPO)

First Logon to a machines that has Group Policy Tools Installed, press Windows + R keys together, type gpedit.msc, and press Enter key.

gpedit-windows-10

Within Group Policy Editor, create a new GPO and link it to the level that you need to link it to. (In my case, the domain level.) Navigate to:

Computer Configuration > Policies > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Licensing.

change remote desktop licensing mode

Next, double-click on the “Use the specified Remote Desktop license servers” setting and then select Enabled option. Finally, enter the names of the license servers (host name or IP address) and then click on the OK button.

use the specified remote desktop license servers

Similarly, double-click on the “Set the Remote Desktop licensing mode” setting and then select Enabled option. Finally, set the licensing mode (Per Device or Per User) and then click on the OK button.

set the remote desktop licensing mode

Once all these changes are done, close Group Policy Editor, go to the RDP Licensing Server and run a gpupdate /force to refresh Group Policy.

Now, when you open the Remote Desktop Licensing Diagnoser, you shouldn’t see any errors like the remote desktop licensing mode is not configured on windows server or any kind of issues regarding your licenses.

Corrected Licensing Diagnoser Result

How To configure Remote Desktop Licensing Mode through PowerShell

You can also use PowerShell to set the Licensing Mode via the Set-RDLicenseConfiguration cmdlet from the RemoteDesktop PowerShell Module which is installed with Remote Desktop Services:


KEEP POSITIVITY ON YOUR SIDE!
CONTINUE TO LEARN!

REFERENCES:
How to Fix Remote Desktop Licensing Mode is Not Configured
Set-RDLicenseConfiguration

Windows Server Core – How to have PowerShell automatically start when logging onto the session.

In my environment, I have a Windows Server (2019) Core edition server installed with Exchange 2019. Most of the time, I have to get on the server to run PowerShell commands for maintenance purposes, etc…

Well, by default, Windows Server Core opens the command prompt when you logon and then I have to manually open PowerShell from there to run cmdlets, etc…

However, if you would like to change the default cmd to PowerShell, you can change it by changing the Registry value.

The Registry that I’m talking is located under the following location:

Change the Shell Value in the Registry

The easiest way I see to change the value is to use the Set-ItemProperty cmdlet within PowerShell.

Open Windows PowerShell within Server Core command prompt. You can type “PowerShell” on your command prompt.

Then, enter the following command on PowerShell console and hit enter:

Once completed, you will need to reboot the computer from PowerShell:

When the computer has rebooted and you have logged on, PowerShell should load by default instead of Command Prompt.

EVEN MORE INFORMATION

Now, since I have an Exchange Server installed on this server, there is a Command in the $bin directory called LaunchEMS.cmd that will load the Exchange Management Shell for you. So instead of loading just PowerShell, I tell WinLogon to load Exchange Management Shell so that I do not have to do any additional typing or searching for EMS on the box. Remember, Server Core has no GUI!

I run the same commands as above, but just change the value to LaunchEMS.cmd

Then Restart the Computer:

Once Rebooted, you can logon and EMS will be the only window prompt that loads in the shell!

Exchange Management Shell loads when you logon

NOTE: You can always run cmd from the prompt to open Command Prompt and also run PowerShell.exe to open regular PowerShell from the EMS Session Window.

REMAIN POSITIVE!
THANKS FOR READING!

REFERENCES:
Windows Server Core: How to start PowerShell by Default

Troubleshooting OOF/OOO (Out of Office) Replies in Exchange and Exchange Online

Since the COVID-19 virus has managed to get me laid-off and not working, I have not had too much to post in the past months. And although it has wavered greatly, I will remain positive and hope that someone will see the value that I can bring to their organization in the near future.

With that said, I wanted to repost this article that was sent to me as this issue arises quite often in the workplace when someone has left for vacation and is not in the office. So, here it is:

Understanding and troubleshooting Out of Office (OOF) replies

Out of Office replies can sometimes be a bit of a mystery to people; how do they work? What do you do if they don’t work? In this blog post, we will discuss the bits and pieces of Out of Office and some of the main reasons why an Out of Office (aka. OOF) reply might not get delivered to users. Note that while we are writing this from the viewpoint of an Exchange Online configuration, many of same things can be applied to on-premises configuration also. By the way – did you ever wonder why “OOF” is used instead of “OOO”? If you did, see this!

What is an Out of Office reply?

Out of Office replies are also known as OOF (or OOO) replies or automatic replies. They are Inbox rules that are set in the user’s mailbox by the client. OOF rules are server-side rules, so the response is sent regardless of whether a client is running, or not.

There are several ways of setting up automatic replies:

First, it can be set up as an automatic reply feature from Outlook, like this. It can also be configured using other clients, such as Outlook on the web (OWA), PowerShell command (Set-MailboxAutoReplyConfiguration). Admins can set up OOF replies on behalf of (forgetful) users from the M365 Admin Portal.

Other than using built-in OOF functionality, another thing people sometimes do is use rules to create an Out of Office message while they are away.

By design, Exchange Online Protection uses the high risk delivery pool (HRDP) to send the out of office replies, because they are lower priority messages.

Types of OOF rules

There are three types of OOF rules: Internal, External and Known Senders (Contact list). They are stored in the mailbox with the names in the following table. If a mailbox has set only Internal OOF, there will be no external rule created and the mailbox will have one OOF rule.

TypeMessage ClassPR_RULE_MSG_NAME
InternalIPM.Rule.Version2.MessageMicrosoft.Exchange.OOF.KnownExternalSenders.Global
ExternalIPM.Rule.Version2.MessageMicrosoft.Exchange.OOF.AllExternalSenders.Global
Known ExternalIPM.ExtendedRule.MessageMicrosoft.Exchange.OOF.KnownExternalSenders.Global

Note: Apart from OOF rule, other rules like the Junk Email rule will also have the “IPM.ExtendedRule.Message”  message class; the MSG_NAME will determine what the rule is for.

OOF rule details

All Inbox rules can be viewed using MFCMapi tool:

Logon > profile that you are accessing > Top of information store > Inbox > right click ‘Open associated contents table’. They are listed under the Message Class column. All Inbox rules will have the same message class IPM.Rule.Version2.Message and there is one message class and name for each Inbox rule.

For all rules, the name of the rule is in stored in the PR_RULE_MSG_NAME property. So, if there are 4 Inbox rules, there will be 4 IPM.Rule.Version2.Message one for each rule, and the name of the rule is stored in PR_RULE_MSG_NAME.

OOF rules in MFCMapi:

OOF01.jpg

And OOF rule templates in MFCMapi:

OOF02.jpg

History of OOF replies

OOF response is sent once per recipient. Recipients to whom the OOF was sent are stored in the OOF history and are cleared out when the OOF state changes (enabled/disabled) or the OOF rule is modified. OOF history is stored in the user’s mailbox and can be seen using MFCMapi tool at: Freebusy Data > PR_DELEGATED_BY_RULE.

OOF03.jpg

Note: If you want to send response to the sender every time instead of just once, you can apply mailbox server side rule “have server reply using a specific message” to send automatic reply instead of using the OOF rule. This server-side rule will send reply to the sender every time a message is received.

Now that we know what OOF replies are and how they are stored on the server, we can move on to address some of the scenarios where OOF is not sent to the sender. We will also discuss possible fixes and some more frequently seen issues you may have with OOF configuration.

The first category of issues we will talk about are issues related to OOF replies not being received by the sender of the original message.

OOF issues related to transport rules

When OOF doesn’t seem to be sent for all users in the tenant, usually there is a transport rule causing the issue. Check all the transport rules that may apply to the affected mailbox using step two of this article.

If you suspect a delivery problem, run a message trace from the Office 365 tenant.  We know that OOF response is sent back to the original sender of the message, so for OOF messages, the sender of the original message becomes the recipient when tracking. We should then be able to tell if OOF reply has been triggered and sent to external or internal recipient. If a Transport rule is blocking the OOF response, the message trace will clearly show you that.

There is one scenario I would like to highlight when it comes to transport rules blocking OOF replies. Let’s assume that you moved the MX record to a 3rd party anti-spam solution; you have created a transport rule to reject any email coming from any other IP address than the 3rd party anti-spam.

The transport rule will look something like this:

Description:
If the message: Is received from ‘Outside the organization’ Take the following actions: reject the message and include the explanation ‘You are not permitted to bypass the MX record!’ with the status code: ‘5.7.1’ Except if the message: sender ip addresses belong to one of these ranges: ‘1xx.1xx.7x.3x’

ManuallyModified: False

SenderAddressLocation: Envelope

As OOF replies have a blank (<>) Return-Path, you will see that the rule is unexpectedly matching the transport rule and the OOF responses are getting blocked.

In order to fix this, you can change the transport rule property of ‘Match sender address in message’ to ‘Header or envelope’, so the checks will also be done against ‘From’ (also known as the ‘Header From’ address), ‘Sender’ or ‘reply-to’ fields. More information about the mail flow rule conditions is here.

OOF04.jpg

JournalingReportNdrTo mailbox setting

If the affected mailbox is the one which is configured under JournalingReportNdrTo, OOF replies will not be sent for that mailbox. Moreover, journaling emails may be affected as well. It is recommended to create a dedicated mailbox for the JournalingReportNdrTo setting. Alternatively, you can set it to an external address.

For more details on how to solve this, please see this KB Article.

Forwarding SMTP address is enabled on the mailbox

If the affected user mailbox has SMTP forwarding enabled, OOF replies won’t be generated. This can be checked in user mailbox settings (OWA):

OOF05.jpg

In PowerShell:

OOF06.jpg

Or, in the M365 Portal, user properties:

OOF07.jpg

Please follow the action on step one of this article for more information.

The type of the OOF reply set on remote domains

Remote domains offer you (among other settings) the opportunity to set the type of OOF reply that can be sent to users.

These types are the following:

  • External
  • ExternalLegacy
  • InternalLegacy
  • None

For more information about each of these OOF types, please refer to AllowedOOFType parameter in our Set-Remotedomain document.

The Out of Office type can be checked from Exchange Admin Center > Mail flow > Remote domains

OOF08.jpg

Or, with PowerShell:

OOF12.jpg

You need pay attention to what OOF type you have set up, as this will impact the OOF response and OOF may not be generated at all if the configuration is incorrect. Let’s assume you have a hybrid organization with mailboxes hosted both in Exchange on-premises and Exchange Online. In this scenario, by design only external messages will be sent to on-premises while AllowedOOFType is set to External. To be able to send internal OOF messages to on-premises in hybrid environment, you need to set the AllowedOOFType to InternalLegacy.

You also have option to send external Out of Office replies only for contacts at the mailbox configuration level (ExternalAudience: Known). This can make automatic replies not being sent to anyone external but contacts. The command to check the configuration is:

OOF10.jpg

Remote domain blocking OOF replies

Another setting on remote domains is one which lets you dictate whether you allow or prevent messages that are automatic replies from client email programs in your organization.

This can be found in Exchange Admin Center > Mail flow > Remote domains

OOF11.jpg

Or by running this PowerShell cmdlet:

OOF12.jpg

Note: If this option is set to false, no automatic replies will be sent to users for that domain. This setting takes precedence over the automatic replies set up at the mailbox level or over the OOF type (discussed above).

Please, keep in mind that $false is the default value for new remote domains that you create as well as the built-in remote domain named Default in Exchange Online.

If the email was marked as spam and sent to junk, an automatic reply will not be generated at all.

Pretty self-explanatory, that one!

Message trace shows delivery failure

If you investigate an OOF reply issue and in the message trace you find the following error message:

“550 5.7.750 Service unavailable. Client blocked from sending from unregistered domains.”

You should reach out to Support to find out why the unregistered domain block was enforced.

There are some other scenarios that might come up when working with OOF replies, let’s cover those next!

An old or duplicate OOF message is sent

This is likely due to a duplicate Inbox rule or the OOF history limit. The OOF history has a limit of 10,000 entries, if this threshold is hit, OOF will continue to be sent to recipients that are not already in the list as any new users can’t be added to the list. All users already in the list will not receive duplicate OOF replies. For more information you may want to check this article or follow the action plan below.

  • Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
  • Disable and then re-enable the OOF feature for the mailbox

Now, you can check again whether the OOF feature works as expected and symptoms do not occur.

Automatic replies cannot be enabled; an error message is received

While attempting to access automatic replies from the Outlook client, an error message is received saying that “Your automatic reply settings cannot be displayed because the server is currently unavailable. Try again later.”

To narrow down this issue, you should perform the following steps:

  • Confirm EWS protocol is enabled on the mailbox as OOF replies rely on it to be enabled (note that re-enabling this might take several hours to take effect)
  • Enable the OOF feature by using the following command:Set-MailboxAutoReplyConfiguration <identity> -AutoReplyState Enabled
  • Check whether the OOF feature works as expected.
  • If the issue is still there, review the rules quota on the mailbox: Get-mailbox -identity <mailbox> | fl RulesQuota
OOF13.jpg

By default, the RulesQuota has a maximum quota which is calculated by the size of the rules (not the number of rules). Maximum is 256 KB (262,144 bytes).

  • Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
    After you remove them, you can re-enable the OOF feature and then test again.

An automatic reply is still sent despite OOF being disabled

We have encountered scenarios in which OOF messages are still being sent, although it is disabled. Most of the time, we found that the rule is created manually by the end users using the out-of-office template.

So, as you can see, there is quite a bit that goes into troubleshooting OOF Replies and it is not all straight forward.

THANKS FOR READING!
I’M AVAILABLE FOR WORK!
PLEASE CONTACT ME FOR AN APPOINTMENT!

REFERENCES:
Troubleshooting OOF Replies (Exchange Team Blog)

PowerShell – How to create a custom view for your PS Output Objects

I have been doing some training on PowerShell Scripting this week and am going to be posting a number of articles on what I have been training on. This article deal with formatting your data output from your custom script or function to be viewed the way that you want it.
Have you ever run a cmdlet where the column width is not wide enough and your data get’s truncated? Well, here is a method that you can use to make the default output of your function or script display how you want it to.

The best way to do this is by using an existing xml formatting file as a template. Run the following PowerShell commands to access those templates:

Note: your custom view xml file will need to have the “.ps1xml” extension, which indicates that it is a “Windows Powershell XML Document”.

Now to creating the custom template. Let’s say you have the following function.

Now, create a custom view using the following file as a template:

C:\WINDOWS\system32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml

Note: You can get a list of the format type template files that already comes with PS running the following command:

Sample Output:

Mode LastWriteTime Length Name
—- ————- —— —-
-a— 10/06/2009 21:41 27338 Certificate.format.ps1xml
-a— 10/06/2009 21:41 27106 Diagnostics.Format.ps1xml
-a— 23/07/2012 19:12 144442 DotNetTypes.format.ps1xml
-a— 23/07/2012 19:12 14502 Event.Format.ps1xml
-a— 23/07/2012 19:12 21293 FileSystem.format.ps1xml
-a— 23/07/2012 19:12 287938 Help.format.ps1xml
-a— 23/07/2012 19:12 97880 HelpV3.format.ps1xml
-a— 23/07/2012 19:12 101824 PowerShellCore.format.ps1xml
-a— 10/06/2009 21:41 18612 PowerShellTrace.format.ps1xml
-a— 23/07/2012 19:12 13659 Registry.format.ps1xml
-a— 23/07/2012 19:12 17731 WSMan.Format.ps1xml

Remember: A custom view must always end with “.format.ps1xml”

We will use DotNetTypes.format.ps1xml as a template. Using this file, create a file called “hrmctools.formatps1xml”. That file will contain the following information modified from the template:

NOTE: In PS, the content of xml files are always CASE SENSITIVE!!!

Once you have created your own custom view you then need to tell PowerShell to apply the formatting by using the cmdlet Update-FormatData:

Once you run the above command, this custom format you created should now be loaded into memory. You can verify this by using the Get-FormatData cmdlet:

If you have not done so in your function or script, you can attach your custom view to your function or script using the “insert” method:

$MyObject.PSObject.TypeNames.Insert(0,’hrmctoolcustomformat’)

Note*: This code has already been inserted into our function listed in this example.
NOTE**:The first parameter “0” is something that you always type in. You can now confirm that the object has successfully been attached to the custom view, by typing in PowerShell:

Also if you output your object, you should now notice that its appearance should have now changed to those that you defined in the custom view.

A “Typename” is essentially a name that you give to your object. It tells PowerShell the type of object that it is. Know that it is possible for a number of commands to have outputting objects of the same type (the same typename value). This could affect other cmdlets and functions in your script, so be sure to debug if necessary.

HAPPY SCRIPTING!
MORE TO COME!!

REFERENCES:
Creating Custom Format Views

Exchange Server Quarterly Updates March 2020

Released: March 2020 Quarterly Exchange Updates

Today Microsoft is announcing the availability of quarterly servicing cumulative updates for Exchange Server 2016 and 2019. These updates include fixes for customer reported issues as well as all previously released security updates. 

Personal Note: I was recently involved in a layoff at Microsoft in the Vendor PFE world. I am currently looking for new engagements.

Calculator Updates

This quarterly Exchange release includes an important update to the Exchange 2019 Sizing Calculator.  We’ve made improvements to the logic to detect whether a design is bound by mailbox size (capacity) or throughput (IOPs) which affects the maximum number of mailboxes a database will support.  Previous versions of the calculator produced incorrect results in some situations.

The Exchange team highly recommends using calculator version 10.4, included with the March 2020 quarterly CU release, to size Exchange Server 2019 deployments.

MCDB Configuration Issues

Cumulative Update 5 for Exchange Server 2019 also fixes an issue that can happen when you use the Manage-MetaCacheDatabase.ps1 script to enable MetaCacheDatabase (MCDB).

This issue occurred because of a change in behavior in Windows Server 2019 that caused Get-Disk to return all uninitialized discs within the Database Availability Group (DAG) or cluster. The script then incorrectly tried to format an SSD on another DAG member. We documented a workaround for CU4 here, but we’ve fixed it in CU5.

Online Mode Search Issues

Cumulative Update 5 for Exchange Server 2019 is also required to fix a known issue with partial word searches when the client is using Outlook in online mode.

Release Details

The KB articles that describe the fixes in each release and product downloads are available as follows:

Additional Information

Microsoft recommends all customers test the deployment of any update in their lab environment to determine the proper installation process for your production environment. For information on extending the schema and configuring Active Directory, please review the appropriate documentation.

Also, to prevent installation issues you should ensure that the Windows PowerShell Script Execution Policy is set to “Unrestricted” on the server being upgraded or installed. To verify the policy settings, run the Get-ExecutionPolicy cmdlet from PowerShell on the machine being upgraded. If the policies are NOT set to Unrestricted you should use the resolution steps in KB981474 to adjust the settings.

Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, or who are using Exchange Online Archiving (EOA) with their on-premises Exchange deployment are required to deploy the currently supported cumulative update for the product version in use, e.g.,

2013 Cumulative Update 23
2016 Cumulative Update 16 or 15
2019 Cumulative Update 5 or 4.

For the latest information on Exchange Server and product announcements please see: 
What’s New in Exchange Server and Exchange Server Release Notes.

I AM STILL CURRENTLY LOOKING FOR A NEW PROJECT OR ASSIGNMENT!
THANKS FOR READING!

Exchange Server Security Update KB4540123 fails with 0x80070643

PLEASE READ THE ENTIRE POST

I had a failure on one of my two Exchange 2019 CU4 servers when installing the Security Update for them:

Exchange Server Security Update KB4540123 fails with 0x80070643

I could not restart the install as it would fail when getting to the services stoppage part of the installation. I saw this error in the ServiceControl.log file in the C:\ExchangeSetupLogs Directory

I had searched around based on the error code 0x80070643 and found these answers that some had used to get the installation to work.

LINK HERE

I downloaded the .msp file to install manually and read the answers on the web-page. It was said that there was an issue with the ServiceControl.ps1 file that is in the Exchange Server BIN directory when running Patch with the Verbose logging enabled:

I kept reading and digging into the fix for the file. It was said to modify a number of lines in the ServiceControl.ps1 script:

Once I made those changes. I renamed the original ServiceContol.ps1 to a .old file and saved this modified file to the BIN directory. I was then able to successfully run the Security Update.

BUT WHY DID THIS SERVER FAIL AND NOT THE OTHER?!?

Both are CU4, but the server that failed had been upgraded from CU1 where the one that did NOT fail was a clean installation of CU4. So, I checked the ServiceConrol.ps1 file on both servers:

Older Exchange Install
Date of ServiceControl.ps1 is 1/1/2020
CU4 Clean Installation
Date of ServiceControl.ps1 is 2/3/2020

I have not tested this, but maybe if I had copied the ServiceControl.ps1 file from the CU4 clean installation to the original install, the script might have worked since the creation dates are different and I have no other version information to go on. I will verify this though. For now, the changes to the script allowed me to successfully install the Security Update Successfully.

NOTE: All the Exchange and IIS Services were in a Startup Mode: Disabled state and I had to reset them ALL to Automatic. Once that was completed and the server rebooted, Exchange was returned to normal state. ****Also, just to be safe, I ran a great script that assures the server is out of maintenance mode. You can get the script HERE. You can also get the script to put the server into maintenance mode. You can get the script HERE. These scripts will work on Exchange 2013 and above servers.

UPDATE / 3/12/2020 2:39 AM EST

Something was still wrong with the server. ActiveSync and PAM started breaking. I started having all sorts of authentication problems. I decided to restore the server from backup from Tuesday. I forgot to backup the ServiceControl.ps1 file that modified, but I moved the one from the successful installation to the restored server and am currently running the update. I will see if it works and send the information to you.

UPDATE / 3/12/2020 3:45 AM EST

The restore worked great and placing the ServiceControl.ps1 file in the BIN directory on the prior failed Exchange Server did allow for the installation to complete successfully. I have tested ActiveSync and Authentication which is now functioning properly. Hooray!

SEND ME YOUR IDEAS/ FOR POSTS!
HAPPY TROUBLESHOOTING!

REFERENCES:
Exchange Server Security Update fails with 0x80070643
Exchange Maintenance Mode Script (Start)
Exchange Maintenance Mode Script (Stop)
Exchange Server 2019 CU4 Security Update

Adding Windows Capability to Server Core to add features needed for Application Compatibility

I’ve been working on installing Windows Server 2019 Core into my network to be able to look at new features for Windows Administration and learning how Server Core works. I was able to install a virtual machine with Server Core and get it activated. I then wanted to place my custom PowerShell script for loading PowerShell into the Server Core Environment.

So, I added the Server Core Server to the Windows Admin Center and copied my custom scripts for PowerShell into the proper directory:

Windows Admin Center
Windows Admin Center

I then logged on remotely to the server and started PowerShell. When I did that, I got this error with the script load:

Error that IE First Run has not been completed
Error that IE First Run has not been completed

At first, I tried using the -UseBasicParsing as a switch to see if that would repair the issue in the script. It did not because, IE is not installed by default on the default installation of Server Core. That is so there is less of a footprint that can be attacked by a hacker. I needed this installed though so that the Invoke-WebRequest cmdlet would load my script parameters properly.

I started looking for answers to how to install IE onto the Server Core box and found the following article. I had to run the Add-WindowsCapability cmdlet on the server to install the optional components. When I did, I received an error:

Error when adding the Windows Capability
Error when adding the Windows Capability

So I found out that there is a block that WSUS does keeping the cmdlet from going to the online source to download the software package and producing this error. After researching, I found this article. I setup a Group Policy to make sure this setting is propagated to my Server Core machine. I also setup in the same policy the ability to turn off the First-Run for IE so that you do not get that message and have to open IE to “set it up”

Group Policy Setting with Path to Templates Specified
Group Policy Setting with Path to Templates Specified

I then ran a gpupdate /force on the Server and was able to download the components for IE and App Compatibility.

Successful Installation of Windows Capability
Successful Installation of Windows Capability

I then rebooted the server and now my PowerShell loads successfully:

Successful PowerShell Load
Successful PowerShell Load

I learned a few different new things here and was able to get Server Core working more the way that I like it. I will keep posting updates when I run into issues with this type of installation. I would definitely give the Windows Admin Center a try as it has more robust features than Server Manager has, especially for Server 2019 and Server Core.

CONQUER THE UNCOMFORTABLE TO GROW!
POSITIVE ATTITUDE ABIDES!

REFERENCES:
RSAT Tools Installation Error 0x800f0954 – Windows 10 1809
Server Core App Compatibility Feature on Demand (FOD)
“Set Up Internet Explorer 11” Bypass with GPO or Registry

Set-SendConnector cmdlet does not function correctly when updating a Send Connector on an Edge Server in an Exchange Hybrid Deployment

I have run into this issue at a number of my customers that utilize an Exchange Edge Server in their Hybrid Deployment. They’ll need to modify their send connectors for their forced TLS communication with their partners or own mailboxes in Office365. Whenever they want to modify the send connector and save the changes, they get the following error messages:

Symptoms

“PowerShell failed to invoke ‘Set-SendConnector’: Error 0x5 (Access is denied) from cli_GetCertificate”

or

“Error 0x6ba (the RPC server is unavailable) from cli_GetCertificate”

This issue occurs after you install the Cumulative Update 14 for Exchange Server 2016Cumulative Update 13 for Exchange Server 2016, or Cumulative Update 23 for Exchange Server 2013.

Cause

This issue occurs because the TLS certificate check (in case the TlsCertificateName attribute is populated on the send connector) doesn’t work against the Edge servers as the RPC communication is blocked against the Edge servers.

Workaround

Now the current workaround for this has been to delete the Edge Send Connector and recreate the connector from scratch via PowerShell with all the settings and changes entered. This is not a viable solution especially if your communications with your partners change constantly and changes are made to the secure communications channel between you and them.

Resolution

To fix this issue, install one of the following updates:

For Exchange Server 2019, install the Cumulative Update 4 for Exchange Server 2019 or a later cumulative update for Exchange Server 2019.

For Exchange Server 2016, install the Cumulative Update 15 for Exchange Server 2016 or a later cumulative update for Exchange Server 2016.

For Exchange Server 2013, there is no fix at this time. My personal recommendation is to plan an upgrade to Exchange 2019.

KEEP POSITIVLY MOVING FORWARD!

REFERENCES
Set-SendConnector doesn’t work for Exchange Server in hybrid scenarios with Edge Server installed

Exchange System Mailboxes not being configured cause Exchange Setup to fail

My continuation of the “Installation from HELL” proceeded onward today with our team attempting to install Exchange on another server in the test environment and having it fail when getting to the Mailbox Role portion of the installation.

The error kept saying that the installation was failing due to a “Database is mandatory on UserMailbox”. We had been having many issues with the Schema and RBAC roles which were resolved in my other post by adding the Role Assignments to the schema. I did mention that the environment started settling down and the system mailboxes (Arbitration) along with the Health Mailboxes started functioning. This was actually not the case for the Arbitration mailboxes. I glanced at the following article to see how to manually recreate the Arbitration mailboxes again.

I performed a “Get-Mailbox -Arbitration | fl Name” in Exchange Powershell (similar to the screenshot below) to see if the mailboxes were in fact created. They in fact were not and were giving the error “Database is mandatory for the UserMailbox.”

Image
Verification of Arbitration System Mailboxes existing

So, I tried to do what the original article said to do and enable the mailboxes one by one. I kept getting errors when trying to create the mailboxes. So I began to search the internet for another way to possibly remediate this without having to get too deep into the system.

I found the following article explaining the exact error I was getting during the installation of Exchange. In the article, it said to look at the attributes of the account associated with the Arbitration mailbox to see if the homeMDB attribute had no value:

Image
homeMDB attribute NOT set on Arbitration Mailbox Account

Now, since I was NOT having good luck with either the Exchange Setup nor PowerShell, I had to figure out a way to place the attribute value so that the mailbox would be visible. What I did was this:

  • I opened a User in ADUC with a working mailbox on the needed database.
  • I went to the Attributes Tab and looked up the homeMDB attribute for that user then chose Edit.
  • I copied the entire value from the screen and closed it.
  • I then went to the Arbitration mailbox in question and opened it’s homeMDB attribute.
  • I pasted the value into the Value box and saved it.
Paste the active database value in the homeMDB attribute for the Arbitration Mailbox account

Once completed with remediating the attribute for all the Arbitration mailbox accounts missing the value, I re-ran the cmdlet to verify that the error was not present for any arbitration maibox:

I then uninstalled and re-installed Exchange using setup on the failing server and the installation completed successfully.

This has been an excellent week in training on the value of the setup process for Exchange and also the value of the system accounts and values in relation to Exchange and it working properly.

A POSITIVE OUTLOOK WILL YIELD POSITIVE RESULTS ULTIMATELY!

REFERENCES:
Exchange Install Error Database is mandatory on UserMailbox
Recreate missing arbitration mailboxes

RBAC Role Assignments NOT installed during Exchange Directory Preparation

I had a very interesting installation issue recently when installing Exchange 2019 into a new environment. We ran through all the Exchange Preparation for the root and child domains in the forest as described HERE. The results of those installation procedures showed SUCCESS, but when we started installing Exchange, we ran into issues with the System Mailboxes not being available to complete the Mailbox Role part of the installation. Most of the articles that I found said to re-run the Domain Prep (/preparealldomains) and the AD Prep (/pad). So we did, and managed to get the first server installed somehow.

The reason I said somehow is because when we tried to logon to the EAC, we would get a 400 Bad Request Error and could not logon to the console. Next, we tried PowerShell and was able to load PowerShell, but I noticed that only ~100 cmdlets loaded. I thought that maybe we had to re-create the account mailbox to get it working properly. Problem was, one of the cmdlets that would not load was Disable-Mailbox along with others like Enable-Mailbox and New-Mailbox. It was as if the admin account we were using had no rights to administer Exchange in any way.

Next, we opened the mailbox in OWA. The mailbox came up okay, so I told the admin to change the URL to /ecp to try and get into the admin center. What happened was that the normal user control panel opened instead, showing again that the account did not have permissions.

We checked replication to the child domain and made sure there were not any apparent AD issues present. There were none. I next started reviewing how Exchange uses RBAC (Role Based Access Control) Groups and Role Assignments to grant users access to Exchange Admin Functionality. I read the following article located HERE.

Something told me to go and check the Schema again, so I went to ADSIEdit > Configuration Container > Services > Microsoft Exchange > (Organization Name) > RBAC > Role Assignments

I looked at the list of role assignments in the window as follows:

Small List of RBAC Role Assignments
RBAC Role Assignments Missing Objects

From the picture, you can see that the list is small, which in my experience is not correct. I verified this by going into my own 2019 environment and comparing the number of objects in that folder:

RBAC Assignments Object list with CORRECT Objects Listed

If you notice the list is MUCH longer and has many more objects listed in the container. So, how did Exchange Setup miss this during preparation? That I will find out later, but first I have to remediate this problem.

CAUSE:

If the RBAC roles assignments are not installed to allow an account to have administrative privileges in Exchange, then you cannot administrate Exchange to even make the necessary changes! Especially so if you’ve only installed ONE server in the environment!

REMEDIATION:

Manually repair the installation by running the script that creates these Objects in the Schema during setup.

******DISCLAIMER: Running the following commands in these instructions, running ADSIEdit, and/or making changes to your Schema and Exchange Installation outside the normal setup process is NOT recommended! Microsoft, LDLNET LLC, nor I (Lance Lingerfelt) are responsible for any issues or errors that may arise from using these instructions, period!******

That said, preform the following to regenerate the objects in the Schema:

1) Open Windows PowerShell (not the Exchange Management Shell) on the server that you installed Exchange Server on with the same account you used to install Exchange.

a. If you have UAC enabled, right click Windows PowerShell and click Run as administrator.

2) Run Start-Transcript c:\RBAC.txt and press Enter

a. This will start logging all commands and output you type to a text file.

3) Run Add-PSSnapin *setup and press Enter

a. This adds the setup snap-in which contains the setup cmdlets used by Exchange during install. You may see errors about loading a format data file. You can ignore those errors.
NOTE: DO NOT run any other cmdlets in this snap-in. Doing so could irreparably damage your Exchange installation.

4) Run Install-CannedRbacRoleAssignments -InvocationMode Install -Verbose and press Enter.

a. This cmdlet should create the required role assignments between the role groups and roles that should have been created during setup.

b. Be sure you run with the Verbose switch so we can capture what the cmdlet does.

5) Run Remove-PSSnapin *setup and press Enter

6) Run $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://servername/PowerShell/ -Authentication Kerberos and press Enter

a. Be sure to replace SERVERNAME with the FQDN of your server.

7) Run Import-PSSession $Session and press Enter

a. You should notice that the normal number of cmdlets load (~700)

8) Run Get-ManagementRoleAssignment and press Enter. If you are able to run the cmdlet, then the remedation worked.

9) Run Stop-Transcript and press Enter

The final check is to return to ADSIEdit and check the container and see if all the objects are there. We also were now able to get into EAC as well as saw that the Arbitration mailboxes were populating along with the Health Mailboxes as needed per the installation.
It was very neat to see how running the Add-PSSnapIn cmdlet opened all the scripts from the Exchange Setup and allowed me to manually fix the installation problem by running the cmdlet script that need to perform that task that setup missed or refused to run.

POST MORTEM REVIEW

I am going to look over the installation logs and see where the installation failed and try to find out why it did not run on the subsequent re-installations of the AD Prep and Domain Prep. I will post those finding in this article when I have that available.
Thanks again to my Microsoft and Trimax teammates for your assistance with this. It has helped the customer in more ways than one!

HAPPY TROUBLESHOOTING!
POSITIVE ATTITUDE YIELDS POSITIVE RESULTS!

Create a custom Windows 10 image for distribution using and ISO image.

I’ve been currently assisting with onboarding at my new Full Time Contractor position at Microsoft. All of the new FTCs received laptops and needed to have the newest build of Windows 10 installed. The issue was that all of our laptops came with Windows 10 Professional and we needed to upgrade them to Enterprise edition.

After finding a working key for Enterprise Edition, we were still having issues joining the MS Azure domain so that we could get all of the needed software properly to being onboarding with Microsoft.

So, after going through a couple of re-images of my laptop with some failures attached to that, I finally was able to get the process down so that time would not be waisted for the oncoming new hires once they received their laptop. The issue was getting the correct build of Windows 10 and getting the proper apps installed in an efficient manner. Since the onboarding process was quickly moving, I needed to find a way to help streamline the process so the others would not have to go through all the mess I went through to get everything setup.

So, I began looking for a way to create a customized ISO for the build that would already have apps, settings, and customizations installed. I found this great article that details the process. I wanted to re-post this article here showing the steps I took to create the customized image by creating a VM in Hyper-V and then converting that completed image to an ISO that could be downloaded and utilized for the installation.

Creating a customized ISO image with pre-installed software and no user accounts

  • A generalized ISO image without any pre-set user accounts, with pre-installed software, desktop, File Explorer and Start customizations will be created.
  • All customizations and personalization will automatically be applied to all new user accounts
  • Clean install will perform a normal OOBE, asking for regional settings, initial user and so on
  • This ISO will be generalized meaning it is hardware independent and can be used to install Windows on any computer capable of running Windows 10, regardless if the machine is a legacy BIOS machine with MBR partitioning, or a UEFI machine with GPT partitioning
  • The ISO image will be bootable on both BIOS / MBR and UEFI / GPT systems

NOTE: This post will show how to use a virtual machine to create the ISO. All virtual machine references and instructions in this tutorial apply to Hyper-V, available in Windows 10 PRO, Education and Enterprise editions. Oracle VirtualBox and VMware users might need to consult their preferred virtualization platform’s documentation if instructions can’t be used as is.
Everything in this instruction can be made in each edition of Windows 10 (in Home and Single Language editions using a third party virtualization platform) with native Windows tools and programs, apart from Windows Deployment and Imaging Tools, part of Windows 10 Assessment and Deployment Kit (ADK) needed later in the post. The ADK is a free native Microsoft tool, downloadable directly from Microsoft.
If you will do this on a Hyper-V virtual machine (which is the recommended method), make sure to set the new virtual machine to use Standard Checkpoints instead of default Production Checkpoints. You can do this in virtual machine’s settings:

Name:  image.png
Views: 109048
Size:  81.9 KB
Use Standard Checkpoints
Virtual machine generation is irrelevant, you can use Generation 1 or 2 as you wish

This method will produce an ISO image which can be compared to any original Windows 10 ISO you download from Microsoft, apart from the fact that it already contains pre-installed software according to your choice. It will also contain a customized and personalized default user profile, the base Windows uses whenever a new user profile will be created.

A customized default user profile means that whenever a new user account is created, all customizations (Start tiles, File Explorer & desktop icon and view settings, colors, wallpaper, theme, screensaver and so on will be applied to new user profile instead of Windows defaults.

Installation using this ISO will take somewhat longer than using a standard ISO because it not only contains full Windows setup, but also the pre-installed software. Notice that depending on how much space pre-installed software takes, you might not be able to burn this ISO to a standard 4.7 GB DVD disk but have to use a dual layer disk or a USB flash drive instead. My customized image came out to be about 8.5 GB in size.

The ISO created will include no user profile folders, personal user data and files.

This ISO image can be used on any hardware setup capable of running Windows and can be shared, subject to people you share the ISO with have valid licenses and / or activation keys for both Windows 10 and pre-installed software.

System Preparation Procedure

  • Download the Windows ISO Installation tool from Microsoft
    • Use this TOOL to download the ISO and create the installation media
  • Install Windows 10 on your VM using the downloaded ISO

NOTE: The normal Windows Download from the link above will download Windows 10 Professional. You will need a key for the installation to upgrade to Enterprise Edition and you will need to be able to activate the copy of Windows to be able to save the customizations you create for your ISO.

  • Boot into Windows 10 and do the following:
    • Activate the Windows Edition your are installing with your key. You will require internet connectivity. I needed Enterprise Edition so I changed the Product Key In Settings to upgrade it from Professional.
    • Install your preferred software, customize and personalize Windows, remove / add Start tiles as you wish, and set your preferred group policies (group policies not available in Home and Single Language editions). Do not run any program you install!
    • Update all software and run Windows Update to get all the latest updates for the image.
    • Notice that software installed now will be included in ISO install media, and will be pre-installed for all users on each computer you install Windows to using this custom ISO.

NOTE: If Windows on your reference machine is not activated, you cannot personalize it. In this case you need to modify Windows theme (wallpaper, screensaver, colors, sounds) as you wish on another, activated Windows 10 machine, save the theme as a theme file, copy it to inactivated reference machine and apply (double click).
Also notice that Edge as well as other UWP apps do not work when signed in to built-in admin account. If you need a browser to download software you have to use a third party browser or Internet Explorer. IE can be started from Run dialog by typing iexplore and clicking OK.

  • Open an elevated command prompt and enter the following:

Windows will now restart in Audit Mode using built-in administrator account. You will see a Sysprep prompt in the middle of display:

Name:  image.png
Views: 107212
Size:  73.1 KB
Sysprep Program Window
Leave it open for now
  • Open Notepad, paste the following code to it, make the necessary changes to customize the installation, and save it as
    File name: unattend.xml (exactly this name!)
    Save as type: All files (important!)
    Save in folder: C:\Windows\System32\Sysprep

  • When Sysprepping with the Generalize switch, which we will soon do, the component CopyProfile being set to be TRUE in answer file has a small issue or rather a small inconvenience: it leaves the last used user folders and recent files of built-in admin to end user’s Quick Access in File Explorer.
  • To fix this, we need to reset Quick Access to default whenever a new user signs in first time. It requires the need to run a small batch file at first logon of new user, and then remove the batch file itself from user’s %appdata% so Quick Access will not be reset on any subsequent logon.
  • To do this, open an elevated (Run as administrator) Notepad (Notepad must be elevated to save in system folders), paste the following code to it, save it as:
    File name: RunOnce.bat (or any name you prefer, with extension .bat)
    Save as type: All files (important!)
    Save File in folder: %appdata%\Microsoft\Windows\Start Menu\Programs\Startup

  • Delete all existing user accounts and their user profile data (Option One in this tutorial)
  • You are currently signed in using Windows built-in administrator account. In File Explorer, open C:\Users\Administrator folder and check that all user folders are empty deleting all possibly found content
  • Run Disk Clean-up, selecting and removing everything possible (tutorial)
  • When the disk has been cleaned, create a checkpoint of the VM from Hyper-V Manager. Right Click VM > Click Checkpoint
Manual Checkpoint from Hyper-V Manager
  • In Sysprep dialog still open on your desktop, select System Cleanup Action: Enter System Out-of-Box Experience (OOBE), select Shutdown Options: Shutdown, select (tick the box) Generalize, click OK:
Name:  image.png
Views: 110675
Size:  14.0 KB
Sysprep Selected Options Before Shutdown

Sysprep will now prepare Windows, shutting down machine when done. LEAVE THE VM OFF AND DO NOT RESTART IT! Now, we continue to the Image Creation section.

Name:  image.png
Views: 110209
Size:  3.4 KB
Sysprep Preparing the Machine

Image Creation Procedure

  • On your Hyper-V Host machine, open Disk Management
  • Select Attach VHD from Action menu:
Name:  image.png
Views: 109944
Size:  18.3 KB
  • Browse to and select your reference virtual machine’s VHD / VHDX file. If you have any checkpoints (AVHD / AVHDX files) created on this vm, select the one with most recent time stamp. Note that you have to select show all files to be able to see checkpoint AVHD / AVHDX files:
Click image for larger version.   Name:	image.png 
Views:	987 
Size:	94.6 KB 
ID:	113169
Select the most recent time stamped file
  • Select the check box labeled Read-only (this is very important!), then click OK:
Name:  image.png
Views: 110148
Size:  15.3 KB
BE SURE TO SELECT READ-ONLY

IMPORTANT: Forgetting to select Read-only will especially when mounting a checkpoint AVHD / AVHDX file make it unusable for Hyper-V! You will NOT be able to boot your VM and could corrupt it should you have write access on the mounted VHDX file.
Windows mounts the virtual hard disk, and all of its partitions, as separate disk. In case of an MBR disk it even mounts the system reserved partition.

Name:  image.png
Views: 110574
Size:  6.3 KB

NOTE: In the above picture the Windows system partition for the reference VM is drive K:

  • Open the Windows system partition VHD to be sure that’s the one where Windows is installed, note the drive letter your host assigned to it.
  • Open an elevated Command Prompt, enter the following command to create a new install.wim file:

NOTE: D:\install.wim path in this case is the drive and directory where you want to save the image file. K:\ path is the capture path with subfolders of the drive you want to image FROM

Click image for larger version.   Name:	image.png 
Views:	907 
Size:	26.5 KB 
ID:	113172
dism command

NOTE: The name given in /name switch in above command is irrelevant as we will name the ISO later on, but is needed for the command to run. Use any name you want to for the switch parameter.
The image process will take time, go get something to eat as I did. On my high end Hyper-V server this takes over 20 minutes, the first half of it without showing any progress indicator whatsoever. DISM works somewhat faster if you don’t use optional switches /checkintegrity and /verify but it is not recommended that you to create install.wim without checking its integrity and verifying it.

  • When completed capturing the image, detach the VHD / VHDX or AVHD / AVHDX file from host by right clicking it in Disk Management and selecting Detach VHD:
Name:  image.png
Views: 109850
Size:  28.5 KB
Detach the VHDX

ISO Image Creation Procedure

  • Mount the original Windows 10 ISO you downloaded in the first step to a Virtuial Drive on your Hyper-V Server Host.
  • Copy its contents (everything) to a folder you create on one of your Hyper-V host’s hard disks:
Click image for larger version.   Name:	image.png 
Views:	1221 
Size:	463.8 KB 
ID:	125114

I named the folder ISO_Files and placed it on the D: drive where I had created the image from the previous section. Alternatively, you can copy the contents of a created Windows 10 install USB or DVD to the ISO_Files​ folder.

  • Browse to your custom install.wim created earlier in previous section. Copy it to Sources folder under the ISO_Files folder, replacing the original install.wim in that directory:
Name:  image.png
Views: 110062
Size:  55.8 KB
Note

IMPORTANT: If the ISO you used when copying the files to the ISO_Files folder has been made with Windows Media Creation Tool, the ISO_Files\Sources folder contains an install.esd file instead of install.wim.
In this case you will naturally not get “File exists” prompt. Simply delete the install.esd file and paste your custom install.wim to replace it.
This will help reduce the overall size of the ISO and not confuse the installation process when ran.

  • Now, we download the latest Windows Assessment and Deployment Kit(ADK)for Windows 10: Windows ADK downloads – Windows Hardware Dev Center
    The full download for the ADK is about 7.5 GB but luckily we only need the Deployment Tools portion. So, unselect everything else except Deployment Tools and click Install:
Name:  image.png
Views: 106320
Size:  348.3 KB
Select Only Deployment Tools for the Installation
  • Once completed, you should have a folder within your start menu for the ADK Tools Installation under the folder Windows Kits. Start the Deployment and Imaging Tools interface program by Running the Program as an Administrator:
Name:  image.png
Views: 110029
Size:  38.8 KB
Right Click the Application and “Run As Administrator”
  • At the command prompt, type cd\ to bring your prompt to the root of the folder path you are on.
  • Type the following command to initiate creation of the ISO image file:

Replace three instances of d:\iso_files with the path to the ISO_Files folder where you copied Windows installation files. Notice that this is not a typo: first two of these instances are typed as argument for switch -b without a space in between the switch and argument. This is to tell the oscdimg command where to find boot files to be added to ISO.

Replace d:\14986PROx64.iso with the path where you want to store the ISO image. This is where you also name the ISO file what you want the file name to be.

Although the command seems a bit complicated, everything in it is needed. For more information about the oscdimg command line options, see: Oscdimg Command-Line Options

Click image for larger version.   Name:	image.png 
Views:	1467 
Size:	54.6 KB 
ID:	113179
Screenshot of the OSCDIMG command ran

You now have a completed ISO image ready for distribution to your machines. The overall process took me about 4 hours to complete with all the customizations that I did. Thanks again to Ten Forums for the article. I have provided references below for your convenience as well.

HAPPY IMAGING!!
PLEASE COMMENT!!

REFERNCES:
Create Windows 10 ISO image from Existing Installation
Open and Use Disk Cleanup in Windows 10
Download Windows 10
Windows 10 sysprep – how to skip entering product key
Windows ADK downloads – Windows Hardware Dev Center

The Windows Time Service, Hyper-V Hosts, and DCs that are VMs.

The sheer craziness of it all! I noticed that my clocks were off on my servers by FOUR minutes. I had originally set in group policy for the PDC emulator for my domain, a VM on one of my Hyper-V hosts, to get the time from the Public NTP hosts. I then configured a group policy to have all the other machines get their time from the PDC Emulator.

This was working great for me until I realized that my Hyper-V hosts were actually controlling the time of the VMs. They were also configured to get the time from the PDC Emulator, but essentially, due to how Hyper-V is configured, the PDC Emulator VM was getting the time from the Host. So, once the time got thrown off, everything went wacky on me!

I’d read through a couple of articles and found the configuration flaw of Hyper-V and the need for those servers to get their time from the external NTP hosts as well as be configured as NTP servers themselves. This totally went against my Group Policy configuration which caused the issue!

Luckily, I had a stand alone server that is a tertiary DC in the domain not running Hyper-V. I was able to get my time synced again properly after performing the following configuration.

  • I had to move the FSMO roles to the tertiary DC with the following cmdlet:

  • I then made sure the tertiary DC was syncing time correctly by running the following on that server:

  • I then removed the Group Policy Object for syncing the time source to the DC that I had linked to my Hyper-V Servers OU in Active Directory
  • Ran a gpupdate /force on the Hyper-V host to remove the policy there
  • I then had to reconfigure the Hyper-V hosts to be NTP Servers and clients that got their time from a public NTP server:

The one problem Hyper-V host that was syncing with the DC VM would not change settings via Group Policy nor through the w32tm cmdlet. I even went into the registry and tried to modify the following keys to make the changes stick:

The values would just not change, most likely due to the time not being synchronized. I had to reboot the server and then run through the process again in order for the changes to stick.

I did look at another article that said to do the following on the DC VM in order for time NOT to sync with the Hyper-V Host:

Go into Hyper-V console on the host machine, right-click on the client VM AD server, and select Settings. Once in here, on the left look under:

Management –> Integration Services
Untick Time Synchronization
Click Apply/OK

Virtual Machine Settings within Hyper-V

Things are running smoothly now. Please view the references at the bottom of the post. There are a couple of great articles about the Time Synchronization process with Hyper-V and why it needs to be setup the way I have it now. I wished I had read it before I originally set this up. I will post the article about getting group policy to handle the time sync process. Just remember, if your PDC Emulator is a VM, don’t sync it to a public NTP server. Sync it to your Hyper-V Host and have the Host sync publicly.
In the long run, I think it is a good design solution to have your Hyper-V hosts time synced to the Public NTP servers than having to remember to configure each VM DC you create to NOT time sync with the host. To each is own though, and one thing I learned from working Microsoft, there are multiple ways to get to the same goal that are technically sound methods.

THANKS FOR READING!
PLEASE COMMENT!

REFERENCES:
Setup of NTP on Hyper-V servers
Time Synchronization in Hyper-V
“It’s Simple!” – Time Configuration in Active Directory
NTP Circular Time Sync – Windows Server 2012 R2 / Hyper-V

Exchange Server Client Access URL Configuration Script

In my career, I have to be able to be efficient as most of my projects are on a time crunch schedule. Being able to quickly configure Exchange when setting up a server environment is crucial to the success of the project.

While still honing my skills in PowerShell, I was attempting to create my own script to help configure all of the Virtual Directories in one shot rather than go to each setting and configure them manually. It did not go very well, so as I do, I research and find great professionals that do great work in scripting so that I may learn from them.

In doing so, I found Paul Cunningham’s script that performs this. I took the following script and modified it to add the PowerShell Virtual Directory to it as I like to configure that as well.

***YOU CAN REM THE LINES OUT SHOULD YOU NOT WANT TO CONFIGURE THAT DIRECTORY***

Here is my version of the script:

NOTES:

  • PowerShell script to configure the Client Access server URLs for Microsoft Exchange Server 2013/2016. All Client Access server URLs will be set to the same namespace.
  • If you are using separate namespaces for each CAS service this script will not handle that.
  • The script sets Outlook Anywhere to use NTLM with SSL required by default.
  • If you have different auth requirements for Outlook Anywhere use the optional parameters to set those.
  • The script sets PowerShell to use Basic with SSL required by default.
  • If you have different authentication requirements for PowerShell use the optional parameters to set those.
  • PowerShell was added to the settings. Please be sure to REM those lines of code should you NOT want to configure the PowerShell Virtual Directory.

USAGE:

HAPPY SCRIPTING!
POSITIVE ENERGY!
PLEASE COMMENT!

REFERENCES:
Exchange Server Client Access URL Configuration Script
PowerShell Script to Configure Exchange Server Client Access URLs

Installing an ‘IP-less’ Exchange Server 2019 Database Availability Group

Yesterday, I posted on how Exchange now uses the Resilient File System (ReFS) to optimize and protect Exchange critical files. Another layer of protection is using a database availability group (DAG) for redundancy and is a necessary factor when designing an Exchange Enterprise Environment.
In this example, I will walk you through the installation of an Exchange Server 2019 DAG as I configured in my environment. This DAG will contain two Exchange Servers in the same site with a third Windows Server 2019 server being the File Share Witness (FSW).

Two Server Exchange DAG Configuration

For my configuration, I configured two identical Windows Server 2019 VMs (same procs, RAM, vhdx drives, partitions, etc…). I configured the Exchange Data Volume using ReFS and mounted them to the same folder on the C: Drive on each server. This is very important for replication to take place successfully when the databases are added to the DAG.


I next went to the Admin server where the FSW would be hosted and added the Exchange Trusted Subsystem Account to the local Administrators group on that server:

IMPORTANT!
Add the Exchange Trusted Subsystem Account to the Local Administrators Group on the FSW.

NOTE: The reason that this is an ‘IP-less’ DAG is that I’m creating a DAG with no cluster administrative access point (CAAP). The DAG has no IP address of its own, and no computer object in Active Directory. The main implication of this is that backup software that relies on the CAAP or backup operations won’t work. This option of an ‘IP-less’ DAG was first introduced in Exchange Server 2013 SP1/CU4, so by now any decent backup products should support this configuration. But you should always verify this with your backup vendor of choice. Also be aware that this is only supported for DAGs that are running on Windows Server 2012 R2 (or later).

Next, we create the DAG from Exchange PowerShell using the New-DatabaseAvailabilityGroup cmdlet. Now remember that since you are using the ReFS system for your database volumes, you will need to specify the -FileSystem parameter within the cmdlet to assure proper setup and replication of the data files.

Next, we add the Exchange Servers that hold the databases that will be replicated within the DAG:

The DAG will now show the two servers as Operational Member Servers:

The FSW Directory was created on the admin01 server when the DAG was created. We can verify that with the following cmdlet:

Next, we add the databases that we want replicated to the DAG as replicated databases. I want all my Databases on EX01 to replicate to EX02 and vice versa for the EX02 Databases. I want the activation preference to remain on the server that the databases were originally created on so I will use the -ActivationPreference parameter to accomplish that. I will go into more detail on Activation Preference in another post.

Now we verify that the Database Copies are healthy on each replication member using the Get-MailboxDatabaseCopyStatus cmdlet. You will see a Healthy Status on the replicated copies:

POSITIVE ENERGY!
KILL NARCISSISM!
HAPPY TROUBLESHOOTING!

REFERENCES:
Installing an Exchange Server 2016 Database Availability Group

Using the Resilient File System for Exchange Server

In my ongoing effort for becoming more knowledgeable on Exchange Server, I found that the preferred new file system for Exchange Databases and Log files is the ReFS.
ReFS is not that new. Microsoft’s Resilient File System (ReFS) was introduced with Windows Server 2012. ReFS is not a direct replacement for NTFS, and is missing some underlying NTFS features, but is designed to be (as the name suggests) a more resilient file system for extremely large amounts of data.

Support for ReFS with Exchange Server

From Exchange Server 2013 and upwards (which includes Exchange Server 2019 today) Microsoft supports the use of ReFS for Exchange servers, and in fact they now recommend it as the preferred file system for Exchange Server 2019, within the following guidelines.

For Exchange Server 2013:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features must be disabled for the database (.edb) files or the entire volume that hosts database files.
  • Hotfix KB2853418 must be installed.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2013 installation volume, but you can consider using ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2016:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.
  • For Windows 2012, the following hotfixes must be installed:

This means that you should continue to use NTFS for your operating system and Exchange Server 2016 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

For Exchange Server 2019:

  • ReFS is supported for volumes containing Exchange database files, log files, and content index files.
  • ReFS is not supported for volumes containing Exchange binaries (the program files).
  • ReFS is not supported for volumes containing the system partition.
  • ReFS data integrity features are recommended to be disabled.

This means that you should continue to use NTFS for your operating system and Exchange Server 2019 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files.

Creating an ReFS Formatted Volume

In Windows Server during the New Volume Wizard when you get to the step for configuring File System Settings change the file system from NTFS to ReFS.

exchange-server-refs

NOTE: Using the New Volume Wizard does not give you the option to disable data integrity at the volume level. To set it at the volume level itself use PowerShell when configuring new volumes. I found this out the hard way and am now re-configuring my volumes to disable the Integrity Streams.

I needed to create the mount point to mount the volume to:

I then got a list of my available disks:

In my case, disk 2 was the one I needed to format and change. I had to create a new partition and then format it:

Once formatted, I mount the volume to the Directory created earlier:

NOTE: Partition 1 on a disk is always reserved for system files on the drive volume. So the active partitions will always start at 2.

Lastly, verify that the partition is online and that the Integrity Streams are turned off:

Additional Considerations

When you are deploying an Exchange 2016 or 2019 DAG and using Autoreseed, the disk reclaimer needs to know which file system to use when formatting spare disks. So when, creating a DAG in Exchange PowerShell, make sure to set the -FileSystem parameter. For Exchange Server 2013 DAGs, manually format the spare volumes with ReFS.

More coming soon. I will post how I setup the “IP-less” DAG for my environment and got replication functional for my Exchange Databases.

REFERENCES:
Exchange 2013 storage configuration options
Exchange 2016 Preferred Architecture
Exchange Storage for Insiders: It’s ESE (Ignite video)
ReFS Exchange Server Volumes
Preparing ReFS Volumes for Exchange

Hyper-V General Access Denied error when trying to load a Virtual Hard Drive and start a VM

I was working on setting up a VM for my server farm and mis-configured one of the vhdx drives. I ended up having to delete that drive and recreate it in Hyper-V manager. When I did though, I received an error stating that I could not start the virtual machine:

An error occurred while attempting to start the selected virtual machine(s).
‘VMName’ failed to start. (Virtual machine ID ‘SomeID’)
‘VMName’ Microsoft Emulated IDE Controller (Instance ID ‘SomeID’): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)
‘VMName’: IDE/ATAPI Account does not have sufficient privilege to open attachment ‘C:\Users\Public\Documents\Hyper-V\Virtual hard disks\DiskName.vhdx’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)
‘VMName’:  Account does not have sufficient privilege to open attachment ‘V:\Hyper-V\Virtual hard disks\DiskName.vhdx’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)

Causes

Each virtual machine is started using a virtual machine account. The virtual machine account needs read and write access to the .vhd/.vhdx file, but if the file has just been copied from somewhere then it most likely lacks the necessary file permissions.
That happened in my case because I had just created the vhdx drive and did not create it from the VM itself. I just attached it to the VM. So, when I booted the VM, it gave the error.

Remediation

There are a few ways that you could remediate the issue. The simplest way, if it is a new VM, is to remove the drive in the VM settings and then re-create it from scratch. That is what fixed it for me.
Another way is to add the VM GUID to the permissions so that it can access the vhdx file properly:

  • If you don’t already have the Hyper-V Manager error dialog open (“An error occurred while attempting to start the selected virtual machine(s) …”) then try to start the virtual machine now. You need the error open.
  • Click “See details”. This will show additional details, and will look something like:

‘PC-Name’ failed to start. (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD) ‘PC-Name’ Microsoft Emulated IDE Controller (Instance ID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
‘PC-Name’: IDE/ATAPI Account does not have sufficient privilege to open attachment ‘E:\Hyper-V\PC-Name\Virtual Hard Disks\MyVHD.vhdx’.
Error: ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
‘PC-Name’: Hyper-V Virtual Machine Management service Account does not have sufficient privolege to open attachment ‘E:\Hyper-V\PC-Name\Virtual Hard Disks\MyVHD.vhdx’.
Error: ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
Where PC-Name will be the name of your virtual PC. The long sequence of letters and numbers (in my case above “B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD”) is the Virtual Machine ID. This number is significant and you need it to fix the problem.

  • On the host server open an elevated command prompt.
  • Enter the following:

You will need to substitute the path to the vhd/vhdx file – you can obtain this from the original error message, and the Virtual-Machine-ID that you obtained from the “See details” part of the error.

So the line for me was:

NOTE: If you get the message “Failed processing 1 files” then check the virtual machine ID.

  • Now try to start the virtual machine. The error should no longer be present.

There is also a PowerShell Gallery script that is supposed to remediate this issue:

http://www.ntsystems.it/page/PS-Restore-VMPermissionps1.aspx

I haven’t tried it but it looks as it would work. Please review and leave a comment should you have issues with the script.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!
POSITIVE ENERGY!

REFERENCES:
Resolved: Hyper-V General access denied error when trying to load a Virtual Hard Drive
Restore-VMPermission
Virtual machine fails to start with General access denied error / Account does not have sufficient privilege to open attachment

Importing User Photos to Office 365 in bulk for your company.

In a previous post, I showed how you could update one user’s photo for their Outlook and AD profiles via PowerShell. In this post, we will explore how to do this for your entire organization via PowerShell to Office365.

NOTE: I have not tested the scripts as I do not have enough mailboxes in my O365 tenant along with not using a ‘.’ in my alias. If the scripts are incorrect, please inform me with the correction and I will update accordingly.

Please make sure that your photos are reviewed before posting, and try to keep the file size of the photos to a minimum. In Office 365, there exists a limitation for the user photo not to be more than 10 KB in size, but I will show you how to get around that limitation.

Having a user photo for each of your users is very beneficial as it personalizes each account to a face in the company. The user photos can be viewed in below locations:

  • Outlook Web Access
  • Contact Card
  • Thumbnail in emails
  • Outlook Client
  • Yammer
  • Lync Client
  • SharePoint (People Search / Newsfeed)

Steps to take:

  1. Remove the 10KB photo size limitation in Exchange Online
  2. Prepare a folder with all users photos
  3. Update the profile photos via a PowerShell cmdlet.

Connect to Exchange Online with the RPS Proxy Method to remove the 10K size limitation

NOTE: In the PowerShell cmdlet above, we connected using a different proxy method. This was to overwrite the limitation of uploading the images with size more than 10KB. Using the different proxy method (/?proxyMethod=RPS ) to connect to Office 365 in the above cmdlet accomplishes this.

Prepare a folder locally and place all the photos in that folder

Create a folder named C:/UserPics and make the filename of each photo be the username of that particular user. (i.e. llingerfelt.png)
The below script should be able account for aliases that have a ‘.’ in the id as well. (i.e. lance.lingerfelt)

NOTE: From my research, there is no set photo type that is required for the photo. My suggestion would be to keep the photos .png for size constraints while maintaining picture clarity.

Update the profile pictures via PowerShell

Create the following script and name it Photos-Update.ps1

Run Photos-Update.ps1 and the script should upload the photos to Office 365 and apply each photo to the corresponding user.

NOTE: If you’re still having some issues with the alias having a ‘.’ in the name, you can also configure the Photos-Update.ps1 script in this manner to get that working properly:

HAPPY SCRIPTING!
PLEASE COMMENT!

store.ldlnet.net
LDLNET LLC! Your Source for Pofessional IT Services!
www.servermonkey.com
Contact ServerMonkey.com for your IT Hardware Needs!

REFERENCES:
How to import Office365 User photos over 10KB & without CSV in bulk

Set the profile pic for a single Exchange user via PowerShell

I wanted to update my picture within my Outlook profile and AD account really quickly without having to go through OWA to do so. I found this cmdlet that will allow for that picture to be changed very quickly via Exchange PowerShell.

NOTE: This can be done with On-Premises Exchange and Exchange Online PowerShell

Old picture within my account

First, download the picture you want to use to the computer that you want to run the cmdlet from. Also, make sure the picture is cropped and centered prior to running the cmdlet. I saved the pic to C:\temp for my scenario. The best format to use would be jpg. I named the file User1_Profile.jpg

Next, open Exchange PowerShell on the computer you saved the pic to and run the following cmdlet to change the photo:

Once completed, the Outlook client should be closed and reopen so that the new picture is visible in the profile.

Picture change completed

I will post how to perform this for multiple users for Exchange and Office365 in a later post.

REFERENCES:
Set User Photo with Exchange PowerShell

Purging Soft Deleted mailboxes from Exchange Server

If you’re a seasoned administrator, you have knowledge that in Exchange, the database settings will allow you to set the deleted mailbox retention. The default is 30 days, but sometimes you need to purge all those deleted mailboxes to do some ‘spring cleaning’ as it were. Note that doing these cmdlets does not change the ‘Whitespace’ of the database or the size. In my case, I had to purge everything of a toxic individual that was tainting my network much to my disappointment and did the following to complete that task.

The following cmdlet will seek all Soft Deleted mailboxes within the database you select and manually purge them from Exchange.

Now, should you only want to remove one mailbox, you will need to get the GUID of that Soft Deleted mailbox first so that you can enter it for the identity parameter.

You can also preform a similar task for a disabled mailbox:

You can perform the task on all disabled mailboxes for that database as well:

NOTE: I would be very careful when performing either of these cmdlets as they will completely purge the mailboxes from the schema. If these cmdlets assist you with your ‘spring cleaning’, I will have been happy to assist.

HAPPY PURGING!
PLEASE COMMENT!
IGNORANCE IS NOT BLISS!

References:
Purging Deleted Mailboxes on Exchange 2013

Server Monkey
LDLNET LLC’s Preferred Server Equipment Hardware Vendor

Installation of Exchange Server 2019 on Windows Server 2019

I have realized recently that I am an Exchange Messaging Professional, but yet, I have not posted the methodology of how I install an Exchange Server Mailbox Role. So here it is!

Install Windows Server 2019

Exchange Server 2019 requires Windows Server 2019 to run. For my environment, I haven’t necessarily need to follow all the enterprise level design aspects of database numbers to mailbox size ratios, number of servers, front/back end configurations, DAG Implementation, etc… If you want or need to delve into that realm, you can go here. I have need for a single server with only a few databases for a small number of mailboxes, so I am approaching it from that standpoint.

So first, in Hyper-V, I configured my VM with the following specifications:

Processors: 2 procs with 2 cores each – 4 Virtual Processors Total
RAM: 32GB with dynamic memory enabled optional
Drives: 2 .vhdx drives of 120GB each (OS / Exchange Data)
CD: Windows 2019 ISO
Default Settings for the rest of the VM Settings

Next I installed Windows Server 2019 Datacenter with the GUI! You can install it on Server Core if you wish. That information can be found in this link.

I ran through the setup of Windows and installed the OS on my first vhdx drive. I booted up, set the local admin password, and logged in. Once in Windows, I went to the Local Server Settings in Server Manager and configured the following settings:

Set the Date, Time, and Time Zone. (Once in the Domain, this would sync through Group Policy)
Set IE ESC to allow Administrators to have full IE access.
Set Remote Desktop Settings to gain RDP access. (This would be locked down with Group Policy as well once on the Domain)
Set the IP Settings to Static Settings. (DNS Servers, Gateway, WINS, etc…)
Join the server to the Active Directory Domain.
Reboot the VM Server.
Logon to your Domain.
Configure Windows Update Settings. (I have WSUS through Group Policy, this was configured automatically upon reboot)
Download and install all Windows Updates for the server. Then Reboot.
Open Disk Management and configure the secondary vhdx drive to be your Exchange Data Drive.
I configured the drive to be a mounted folder ‘C:\Exchange\Data’ rather than another drive letter as that seems to be the more accepted form of installation for the data drive these days. That is based on the multiple configurations that I have seen for Exchange through experience in Enterprise environments. Again, to each is own and depending on you design specifications, you might want to do that differently.

Next, we need to install the prerequisites for the Exchange Mailbox Server. I have always used practical365.com to get the PowerShell script to install the prerequisites, but couldn’t find the article this time. Great site though! Instead, I got the information and ran the following from an elevated PowerShell Session locally on the server:

As part of the prerequisites you will need to install the following packages onto the server as well:

UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013

Once completed, you can begin the install of Exchange. If this is your first Exchange 2019 Server in your Organization, then you will need to run the following to update the Forest, Schema, and Domain so that Exchange will install properly:

NOTE: If you run into Prerequisite issues with the installation due to a “pending reboot”, check out my blog post for information on remediation of that issue.

Now that the environment is prepared for Exchange, you can actually begin the installation. I wanted to make my default database and logs folder to be on the Exchange Data volume that I created, so I included those settings in the setup command. Please look at the reference to the setup.exe switches for more information on that. Here is the command:

Setup should go through the installation via the PowerShell window and complete successfully. Reboot the Exchange Server, then you can then logon to the Exchange Admin Center and begin the process of configuration of how you need to integrate the Mailbox Server into your Server Farm. That configuration is for a later post.

PLEASE CHECK BACK FOR UPDATES!
PLEASE COMMENT!

References:
UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013
Install Exchange Server 2019 on Windows Server 2019 Core
Exchange Server Design Planning
Use unattended mode in Exchange Setup
Practical365 on Exchange 2019