Installation and Configuration of Azure Information Protection Unified Labels Scanner

With the release of Unified Labeling in Azure and M365, there is now a way to protect your data and label your data appropriately for confidentiality and encryption for your files shares and files on your on premises devices. The following shows how to install the latest AIP_UL client and configure it in Azure to apply those Unified Labeling Policies.

This is a detailed process and I had some issues myself with getting the process simplified. I will do my best here to make this as smoot has possible with as many reference documents that I can input. Always feel free to comment as this data is ever changing and updating as Microsoft updates the offering.

Prerequisites

Please refer to this document for a full list of pre-requisites before deploying the scanner:

https://docs.microsoft.com/en-us/azure/information-protection/deploy-aip-scanner-prereqs

For the basics we have the following:

The prerequisites below are still required for successful AIP scanner installation.

  • A Windows Server 2012 R2 or greater Server to run the service
    • Minimum 4 CPU and 4GB RAM physical or virtual.
      NOTE: More RAM is better. The scanner will allocate RAM 2.5-3 times of size of all files being scanned in parallel. Thus, if you scan 40 files that are 20MB each at the same time, it should take about 202.540=2GB RAM. However, if you have one big 1GB file it can take 3GB of RAM just for that file.
  • Internet connectivity necessary for Azure Information Protection
  • A SQL Server 2012+ local or remote instance (Any version from Express or better is supported)
    • Sysadmin role needed to install scanner service (the user running Install-AIPScanner, not the service account)

      NOTE: If using SQL Server Express, the SQL Instance name is ServerName\SQLExpress.

      NOTE: At this time, a different SQL instance is needed for each AIP Scanner node.
  • Service account created in On Premises AD (I will call this account AIPScanner in this document).
    • Service requires Log on locally right and Log on as a service right (the second will be given during scanner service install).
    • Service account requires Read permissions to each repository for discovery and Read/Write permissions for classification/protection.
  • AzInfoProtection_UL.exe is available on the Microsoft Download Center (The scanner bits are included with the AIP Client)
  • The Azure AD Preview PowerShell module. From the machine you’re installing AIP Scanner on, run the following from an Administrator PowerShell:

Configure the scanner in the Azure portal

Before you install the scanner, or upgrade it from an older general availability version, configure or verify your scanner settings in the Azure Information Protection area of the Azure portal.

To configure your scanner:

  1. Sign in to the Azure portal with one of the following roles:
    • Compliance administrator
    • Compliance data administrator
    • Security administrator
    • Global administrator

      Then, navigate to the Azure Information Protection pane. For example, in the search box for resources, services, and docs, start typing Information and select Azure Information Protection.
  2. Create a scanner cluster. This cluster defines your scanner and is used to identify the scanner instance, such as during installation, upgrades, and other processes.
  3. Scan your network for risky repositories. Create a network scan job to scan a specified IP address or range, and provide a list of risky repositories that may contain sensitive content you’ll want to secure. Run your network scan job and then analyze any risky repositories found.
  4. Create a content scan job to define the repositories you want to scan.

Create a scanner cluster

  1. From the Scanner menu on the left, select Clusters clusters icon.
  2. On the Azure Information Protection – Clusters pane, select Add add icon.
  3. On the Add a new cluster pane, enter a meaningful name for the scanner, and an optional description. The cluster name is used to identify the scanner’s configurations and repositories. For example, you might enter Europe to identify the geographical locations of the data repositories you want to scan. You’ll use this name later on to identify where you want to install or upgrade your scanner.
  4. Select Save save icon to save your changes.
  5. On the Add a new cluster pane, enter a meaningful name for the scanner, and an optional description. The cluster name is used to identify the scanner’s configurations and repositories. For example, you might enter Europe to identify the geographical locations of the data repositories you want to scan. You’ll use this name later on to identify where you want to install or upgrade your scanner.
  6. Select Save save icon to save your changes.

Create a network scan job (public preview)

Starting in version 2.8.85.0, you can scan your network for risky repositories. Add one or more of the repositories found to a content scan job to scan them for sensitive content.

Note: The network discovery interface is currently in gradual deployment and will be available in all regions by September 15, 2020.

Network discovery prerequisites

PrerequisiteDescription
Install the Network Discovery serviceIf you’ve recently upgraded your scanner, you may need to still install the Network Discovery service.

Run the Install-MIPNetworkDiscovery cmdlet to enable network scan jobs.
Azure Information Protection analyticsMake sure that you have Azure Information Protection analytics enabled.

In the Azure portal, go to Azure Information Protection > Manage > Configure analytics (Preview).

For more information, see Central reporting for Azure Information Protection (public preview).

Creating a network scan job

  1. Log in to the Azure portal, and go to Azure Information Protection. Under the Scanner menu on the left, select Network scan jobs (Preview) network scan jobs icon.
  2. On the Azure Information Protection – Network scan jobs pane, select Add add icon.
  3. On the Add a new network scan job page, define the following settings:

    Network scan job name: Enter a meaningful name for this job. This field is required.
    Description: Enter a meaningful description.
    Select the cluster: From the dropdown, select the cluster you want to use to scan the configured network locations.

    Tip: When selecting a cluster, make sure that the nodes in the cluster you assign can access the configured IP ranges via SMB.

    Configure IP ranges to discover: Click to define an IP address or range.

    In the Choose IP ranges pane, enter an optional name, and then a start IP address and end IP address for your range.

    Tip: To scan a specific IP address only, enter the identical IP address in both the Start IP and End IP fields.

    Set schedule: Define how often you want this network scan job to run.

    If you select Weekly, the Run network scan job on setting appears. Select the days of the week where you want the network scan job to run.
    Set start time (UTC): Define the date and time that you want this network scan job to start running. If you’ve selected to run the job daily, weekly, or monthly, the job will run at the defined time, at the recurrence you’ve selected.

    Note: Be careful when setting the date to any days at the end of the month. If you select 31, the network scan job will not run in any month that has 30 days or fewer.
  4. Select Save save icon to save your changes.

 Tip: If you want to run the same network scan using a different scanner, change the cluster defined in the network scan job. Return to the Network scan jobs pane, and select Assign to cluster to select a different cluster now, or Unassign cluster to make additional changes later.

Analyze risky repositories found

Repositories found, either by a network scan job, a content scan job, or by user access detected in log files, are aggregated and listed on the Scanner > Repositories repositories icon pane.

If you’ve defined a network scan job and have set it to run at a specific date and time, wait until it’s finished running to check for results. You can also return here after running a content scan job to view updated data.

  1. Under the Scanner menu on the left, select Repositories repositories icon.
    The repositories found are shown as follows:
    • The Repositories by status graph shows how many repositories are already configured for a content scan job, and how many are not.
    • The Top 10 unmanaged repositories by access graph lists the top 10 repositories that are not currently assigned to a content scan job, as well as details about their access levels. Access levels can indicate how risky your repositories are.
  2. Do any of the following:
    columns icon Select Columns to change the table columns displayed.
    refresh icon If your scanner has recently run network scan results, select Refresh to refresh the page.
    add icon Select one or more repositories listed in the table, and then select Assign Selected Items to assign them to a content scan job.
    Filter The filter row shows any filtering criteria currently applied. Select any of the criteria shown to modify its settings, or select Add Filter to add new filtering criteria. Select Filter to apply your changes and refresh the table with the updated filter.
    Log Analytics icon In the top-right corner of the unmanaged repositories graph, click the Log Analytics icon to jump to Log Analytics data for these repositories.

Repositories with public access

Repositories where Public access is found to have read or read/write capabilities may have sensitive content that must be secured. If Public access is false, the repository not accessible by the public at all.

Public access to a repository is only reported if you’ve set a weak account in the StandardDomainsUserAccount parameter of the Install-MIPNetworkDiscovery or Set-MIPNetworkDiscovery cmdlets.

  • The accounts defined in these parameters are used to simulate the access of a weak user to the repository. If the weak user defined there can access the repository, this means that the repository can be accessed publicly.
  • To ensure that public access is reported correctly, make sure that the user specified in these parameters is a member of the Domain Users group only.

Create a content scan job

Deep dive into your content to scan specific repositories for sensitive content.

You may want to do this only after running a network scan job to analyze the repositories in your network, but can also define your repositories yourself.

  1. Under the Scanner menu on the left, select Content scan jobs.
  2. On the Azure Information Protection – Content scan jobs pane, select Add add icon.
  3. For this initial configuration, configure the following settings, and then select Save but do not close the pane.

    Content scan job settings
    – Schedule: Keep the default of Manual
    – Info types to be discovered: Change to Policy only
    – Configure repositories: Do not configure at this time because the content scan job must first be saved.

    Policy enforcement
    Enforce: Select Off
    – Label files based on content: Keep the default of On
    – Default label: Keep the default of Policy default
    – Relabel files: Keep the default of OffConfigure file settings– Preserve “Date modified”, “Last modified” and “Modified by”: Keep the default of On
    – File types to scan: Keep the default file types for Exclude
    – Default owner: Keep the default of Scanner Account
  4. Now that the content scan job is created and saved, you’re ready to return to the Configure repositories option to specify the data stores to be scanned. Specify UNC paths, and SharePoint Server URLs for SharePoint on-premises document libraries and folders. 

    Note: SharePoint Server 2019, SharePoint Server 2016, and SharePoint Server 2013 are supported for SharePoint. SharePoint Server 2010 is also supported when you have extended support for this version of SharePoint.

    To add your first data store, while on the Add a new content scan job pane, select Configure repositories to open the Repositories pane:Configure data repositories for the Azure Information Protection scanner
    1. On the Repositories pane, select Add:

      Add data repository for the Azure Information Protection scanner
    2. On the Repository pane, specify the path for the data repository, and then select Save.
      • For a network share, use \\Server\Folder.
      • For a SharePoint library, use http://sharepoint.contoso.com/Shared%20Documents/Folder.
      • For a local path: C:\Folder
      • For a UNC path: \\Server\Folder 

        Note: Wildcards are not supported and WebDav locations are not supported.
    3. If you add a SharePoint path for Shared Documents:
      • Specify Shared Documents in the path when you want to scan all documents and all folders from Shared Documents.
        For example: http://sp2013/SharedDocuments
      • Specify Documents in the path when you want to scan all documents and all folders from a subfolder under Shared Documents.
        For example: http://sp2013/Documents/SalesReports
        or, specify only the FQDN of your SharePoint,
        For example http://sp2013 to discover and scan all SharePoint sites and subsites under a specific URL and subtitles under this URL.
      • Grant scanner Site Collector Auditor rights to enable this.
      • For the remaining settings on this pane, do not change them for this initial configuration, but keep them as Content scan job default. The default setting means that the data repository inherits the settings from the content scan job.

        Use the following syntax when adding SharePoint paths:
        Root path: http://<SharePoint server name>

        Scans all sites, including any site collections allowed for the scanner user.
        Requires additional permissions to automatically discover root content

        Specific SharePoint subsite or collection:
        One of the following:
        – http://<SharePoint server name>/<subsite name>
        – http://SharePoint server name>/<site collection name>/<site name>


        Requires additional permissions to automatically discover site collection content

        Specific SharePoint library:
        One of the following:
        – http://<SharePoint server name>/<library name>
        – http://SharePoint server name>/.../<library name>


        Specific SharePoint folder: http://<SharePoint server name>/.../<folder name>
  5. Repeat the previous steps to add as many repositories as needed.
  6. When you’re done, close both the Repositories and Content scan job panes.

Back on the Azure Information Protection – Content scan job pane, your content scan name is displayed, together with the SCHEDULE column showing Manual and the ENFORCE column is blank.

You’re now ready to install the scanner with the content scanner job that you’ve created. Continue with Scanner Installation.

Scanner Installation

Now that we have verified that all prerequisites and configured the AIP in Azure, we can go through the basic scanner install.

  1. Log onto the server where you will install the AIP Scanner service using an account that is a local administrator of the server and has permission to write to the SQL Server master database. (more restrictive scenarios are documented in the official documentation)
  2. Run AzInfoProtection_UL.exe on the server and step through the client install (this also drops the AIP Scanner bits).
    WARNING: This blog is based on the current version of the AIP Client.  If you want to update to the Preview client, please install the GA first and then install the preview client and use Update-AIPScanner after installation.
  3. Next, open an Administrative PowerShell prompt.
  4. At the PowerShell prompt, type the following command and press Enter:

undefined
Output from Scanner Installation in PowerShell

Create Cloud Service Account

If you are not using Azure AD Sync for your Service account, you will need to create a service account in the cloud tenant to use for AIP authentication. If you have synced your on premises service account, you can skip this task.

  1. Run the command below to connect to Azure AD.

  1. When prompted, provide tenant Global Admin credentials.
  2. To create an account in the cloud, you must first define a password profile object. Run the commands below to define this object.

  1. When prompted, enter a password for the cloud service account.
  2. To create the account, run the commands below.

  1. When prompted, enter the tenant name you want to use for the UserPrincipalName for the cloud service account (e.g. tenant.onmicrosoft.com).

Creating the Azure AD Application in Azure

Next, we will configure the App Registration for the Web App that is required to run the Set-AIPAuthentication command that will be used to get the authentication token. We will also assign the necessary Oauth2Permissions for the Web App to have delegated rights to the App.

  1. Run the commands below to create the Web App, associated Service Principal, and key password.

  1. Next, we need to run some commands to build the RequiredResourceAccess object that is needed to automate delegation of permissions for the native application.

  1. Now we can create the App and associated Service Principal using the commands below.

Authenticating as the AIP Scanner Service

In this task, we will use the command created previously to authenticate the AIP Scanner to the AIP Service.

  1. Open PowerShell using Run as a different user and use the on premises Scanner Service account which should have Run As Administrator rights.

    undefined
  1. Run the commands in the following PowerShell session with the Run as Administrator option, which is required for the OnBehalfOf parameter.
    • The first command creates a PSCredential object and stores the specified Windows user name and password in the $pscreds variable. When you run this command, you are prompted for the password for the user name that you specified.
    • The second command acquires an access token that is combined with the application so that the token becomes valid for 1 year, 2 years, or never expires, according to your configuration of the registered app in Azure AD. The user name of scanner@contoso.com sets the user context to download labels and label policies from your labeling management center, such as the Office 365 Security & Compliance Center.

Successful OUTPUT:
Acquired application access token on behalf of DOMAIN\scanner

  1. Last Step is to Restart the AIP Scanner Service

Look to these reference documents for further details:
Set-AIPAuthentication
Get Azure AD Token for the AIP Scanner

Configure the scanner to apply classification and protection

The default settings configure the scanner to run once, and in reporting-only mode.

To change these settings, edit the content scan job:

  1. In the Azure portal, on the Azure Information Protection – Content scan jobs pane, select the cluster and content scan job to edit it.
  2. On the Content scan job pane, change the following, and then select Save:
    • From the Content scan job section: Change the Schedule to Always
    • From the Policy enforcement section: Change Enforce to On 

      Tip: You may want to change other settings on this pane, such as whether file attributes are changed and whether the scanner can relabel files. Use the information popup help to learn more information about each configuration setting.
  3. Make a note of the current time and start the scanner again from the Azure Information Protection – Content scan jobs pane:

    Initiate scan for the Azure Information Protection scanner

    Alternatively, run the following command in your PowerShell session:

The scanner is now scheduled to run continuously. When the scanner works its way through all configured files, it automatically starts a new cycle so that any new and changed files are discovered.

MUCH MORE TO COME! CHECK OFTEN AND SEND COMMENTS!

REFERENCES:
AIP Prerequisites
Install and Configure AIP UL Application
AIP UL Client Download
AIP Classic Client Express Installation

Windows Server Core – How to have PowerShell automatically start when logging onto the session.

In my environment, I have a Windows Server (2019) Core edition server installed with Exchange 2019. Most of the time, I have to get on the server to run PowerShell commands for maintenance purposes, etc…

Well, by default, Windows Server Core opens the command prompt when you logon and then I have to manually open PowerShell from there to run cmdlets, etc…

However, if you would like to change the default cmd to PowerShell, you can change it by changing the Registry value.

The Registry that I’m talking is located under the following location:

Change the Shell Value in the Registry

The easiest way I see to change the value is to use the Set-ItemProperty cmdlet within PowerShell.

Open Windows PowerShell within Server Core command prompt. You can type “PowerShell” on your command prompt.

Then, enter the following command on PowerShell console and hit enter:

Once completed, you will need to reboot the computer from PowerShell:

When the computer has rebooted and you have logged on, PowerShell should load by default instead of Command Prompt.

EVEN MORE INFORMATION

Now, since I have an Exchange Server installed on this server, there is a Command in the $bin directory called LaunchEMS.cmd that will load the Exchange Management Shell for you. So instead of loading just PowerShell, I tell WinLogon to load Exchange Management Shell so that I do not have to do any additional typing or searching for EMS on the box. Remember, Server Core has no GUI!

I run the same commands as above, but just change the value to LaunchEMS.cmd

Then Restart the Computer:

Once Rebooted, you can logon and EMS will be the only window prompt that loads in the shell!

Exchange Management Shell loads when you logon

NOTE: You can always run cmd from the prompt to open Command Prompt and also run PowerShell.exe to open regular PowerShell from the EMS Session Window.

REMAIN POSITIVE!
THANKS FOR READING!

REFERENCES:
Windows Server Core: How to start PowerShell by Default

Troubleshooting OOF/OOO (Out of Office) Replies in Exchange and Exchange Online

Since the COVID-19 virus has managed to get me laid-off and not working, I have not had too much to post in the past months. And although it has wavered greatly, I will remain positive and hope that someone will see the value that I can bring to their organization in the near future.

With that said, I wanted to repost this article that was sent to me as this issue arises quite often in the workplace when someone has left for vacation and is not in the office. So, here it is:

Understanding and troubleshooting Out of Office (OOF) replies

Out of Office replies can sometimes be a bit of a mystery to people; how do they work? What do you do if they don’t work? In this blog post, we will discuss the bits and pieces of Out of Office and some of the main reasons why an Out of Office (aka. OOF) reply might not get delivered to users. Note that while we are writing this from the viewpoint of an Exchange Online configuration, many of same things can be applied to on-premises configuration also. By the way – did you ever wonder why “OOF” is used instead of “OOO”? If you did, see this!

What is an Out of Office reply?

Out of Office replies are also known as OOF (or OOO) replies or automatic replies. They are Inbox rules that are set in the user’s mailbox by the client. OOF rules are server-side rules, so the response is sent regardless of whether a client is running, or not.

There are several ways of setting up automatic replies:

First, it can be set up as an automatic reply feature from Outlook, like this. It can also be configured using other clients, such as Outlook on the web (OWA), PowerShell command (Set-MailboxAutoReplyConfiguration). Admins can set up OOF replies on behalf of (forgetful) users from the M365 Admin Portal.

Other than using built-in OOF functionality, another thing people sometimes do is use rules to create an Out of Office message while they are away.

By design, Exchange Online Protection uses the high risk delivery pool (HRDP) to send the out of office replies, because they are lower priority messages.

Types of OOF rules

There are three types of OOF rules: Internal, External and Known Senders (Contact list). They are stored in the mailbox with the names in the following table. If a mailbox has set only Internal OOF, there will be no external rule created and the mailbox will have one OOF rule.

TypeMessage ClassPR_RULE_MSG_NAME
InternalIPM.Rule.Version2.MessageMicrosoft.Exchange.OOF.KnownExternalSenders.Global
ExternalIPM.Rule.Version2.MessageMicrosoft.Exchange.OOF.AllExternalSenders.Global
Known ExternalIPM.ExtendedRule.MessageMicrosoft.Exchange.OOF.KnownExternalSenders.Global

Note: Apart from OOF rule, other rules like the Junk Email rule will also have the “IPM.ExtendedRule.Message”  message class; the MSG_NAME will determine what the rule is for.

OOF rule details

All Inbox rules can be viewed using MFCMapi tool:

Logon > profile that you are accessing > Top of information store > Inbox > right click ‘Open associated contents table’. They are listed under the Message Class column. All Inbox rules will have the same message class IPM.Rule.Version2.Message and there is one message class and name for each Inbox rule.

For all rules, the name of the rule is in stored in the PR_RULE_MSG_NAME property. So, if there are 4 Inbox rules, there will be 4 IPM.Rule.Version2.Message one for each rule, and the name of the rule is stored in PR_RULE_MSG_NAME.

OOF rules in MFCMapi:

OOF01.jpg

And OOF rule templates in MFCMapi:

OOF02.jpg

History of OOF replies

OOF response is sent once per recipient. Recipients to whom the OOF was sent are stored in the OOF history and are cleared out when the OOF state changes (enabled/disabled) or the OOF rule is modified. OOF history is stored in the user’s mailbox and can be seen using MFCMapi tool at: Freebusy Data > PR_DELEGATED_BY_RULE.

OOF03.jpg

Note: If you want to send response to the sender every time instead of just once, you can apply mailbox server side rule “have server reply using a specific message” to send automatic reply instead of using the OOF rule. This server-side rule will send reply to the sender every time a message is received.

Now that we know what OOF replies are and how they are stored on the server, we can move on to address some of the scenarios where OOF is not sent to the sender. We will also discuss possible fixes and some more frequently seen issues you may have with OOF configuration.

The first category of issues we will talk about are issues related to OOF replies not being received by the sender of the original message.

OOF issues related to transport rules

When OOF doesn’t seem to be sent for all users in the tenant, usually there is a transport rule causing the issue. Check all the transport rules that may apply to the affected mailbox using step two of this article.

If you suspect a delivery problem, run a message trace from the Office 365 tenant.  We know that OOF response is sent back to the original sender of the message, so for OOF messages, the sender of the original message becomes the recipient when tracking. We should then be able to tell if OOF reply has been triggered and sent to external or internal recipient. If a Transport rule is blocking the OOF response, the message trace will clearly show you that.

There is one scenario I would like to highlight when it comes to transport rules blocking OOF replies. Let’s assume that you moved the MX record to a 3rd party anti-spam solution; you have created a transport rule to reject any email coming from any other IP address than the 3rd party anti-spam.

The transport rule will look something like this:

Description:
If the message: Is received from ‘Outside the organization’ Take the following actions: reject the message and include the explanation ‘You are not permitted to bypass the MX record!’ with the status code: ‘5.7.1’ Except if the message: sender ip addresses belong to one of these ranges: ‘1xx.1xx.7x.3x’

ManuallyModified: False

SenderAddressLocation: Envelope

As OOF replies have a blank (<>) Return-Path, you will see that the rule is unexpectedly matching the transport rule and the OOF responses are getting blocked.

In order to fix this, you can change the transport rule property of ‘Match sender address in message’ to ‘Header or envelope’, so the checks will also be done against ‘From’ (also known as the ‘Header From’ address), ‘Sender’ or ‘reply-to’ fields. More information about the mail flow rule conditions is here.

OOF04.jpg

JournalingReportNdrTo mailbox setting

If the affected mailbox is the one which is configured under JournalingReportNdrTo, OOF replies will not be sent for that mailbox. Moreover, journaling emails may be affected as well. It is recommended to create a dedicated mailbox for the JournalingReportNdrTo setting. Alternatively, you can set it to an external address.

For more details on how to solve this, please see this KB Article.

Forwarding SMTP address is enabled on the mailbox

If the affected user mailbox has SMTP forwarding enabled, OOF replies won’t be generated. This can be checked in user mailbox settings (OWA):

OOF05.jpg

In PowerShell:

OOF06.jpg

Or, in the M365 Portal, user properties:

OOF07.jpg

Please follow the action on step one of this article for more information.

The type of the OOF reply set on remote domains

Remote domains offer you (among other settings) the opportunity to set the type of OOF reply that can be sent to users.

These types are the following:

  • External
  • ExternalLegacy
  • InternalLegacy
  • None

For more information about each of these OOF types, please refer to AllowedOOFType parameter in our Set-Remotedomain document.

The Out of Office type can be checked from Exchange Admin Center > Mail flow > Remote domains

OOF08.jpg

Or, with PowerShell:

OOF12.jpg

You need pay attention to what OOF type you have set up, as this will impact the OOF response and OOF may not be generated at all if the configuration is incorrect. Let’s assume you have a hybrid organization with mailboxes hosted both in Exchange on-premises and Exchange Online. In this scenario, by design only external messages will be sent to on-premises while AllowedOOFType is set to External. To be able to send internal OOF messages to on-premises in hybrid environment, you need to set the AllowedOOFType to InternalLegacy.

You also have option to send external Out of Office replies only for contacts at the mailbox configuration level (ExternalAudience: Known). This can make automatic replies not being sent to anyone external but contacts. The command to check the configuration is:

OOF10.jpg

Remote domain blocking OOF replies

Another setting on remote domains is one which lets you dictate whether you allow or prevent messages that are automatic replies from client email programs in your organization.

This can be found in Exchange Admin Center > Mail flow > Remote domains

OOF11.jpg

Or by running this PowerShell cmdlet:

OOF12.jpg

Note: If this option is set to false, no automatic replies will be sent to users for that domain. This setting takes precedence over the automatic replies set up at the mailbox level or over the OOF type (discussed above).

Please, keep in mind that $false is the default value for new remote domains that you create as well as the built-in remote domain named Default in Exchange Online.

If the email was marked as spam and sent to junk, an automatic reply will not be generated at all.

Pretty self-explanatory, that one!

Message trace shows delivery failure

If you investigate an OOF reply issue and in the message trace you find the following error message:

“550 5.7.750 Service unavailable. Client blocked from sending from unregistered domains.”

You should reach out to Support to find out why the unregistered domain block was enforced.

There are some other scenarios that might come up when working with OOF replies, let’s cover those next!

An old or duplicate OOF message is sent

This is likely due to a duplicate Inbox rule or the OOF history limit. The OOF history has a limit of 10,000 entries, if this threshold is hit, OOF will continue to be sent to recipients that are not already in the list as any new users can’t be added to the list. All users already in the list will not receive duplicate OOF replies. For more information you may want to check this article or follow the action plan below.

  • Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
  • Disable and then re-enable the OOF feature for the mailbox

Now, you can check again whether the OOF feature works as expected and symptoms do not occur.

Automatic replies cannot be enabled; an error message is received

While attempting to access automatic replies from the Outlook client, an error message is received saying that “Your automatic reply settings cannot be displayed because the server is currently unavailable. Try again later.”

To narrow down this issue, you should perform the following steps:

  • Confirm EWS protocol is enabled on the mailbox as OOF replies rely on it to be enabled (note that re-enabling this might take several hours to take effect)
  • Enable the OOF feature by using the following command:Set-MailboxAutoReplyConfiguration <identity> -AutoReplyState Enabled
  • Check whether the OOF feature works as expected.
  • If the issue is still there, review the rules quota on the mailbox: Get-mailbox -identity <mailbox> | fl RulesQuota
OOF13.jpg

By default, the RulesQuota has a maximum quota which is calculated by the size of the rules (not the number of rules). Maximum is 256 KB (262,144 bytes).

  • Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
    After you remove them, you can re-enable the OOF feature and then test again.

An automatic reply is still sent despite OOF being disabled

We have encountered scenarios in which OOF messages are still being sent, although it is disabled. Most of the time, we found that the rule is created manually by the end users using the out-of-office template.

So, as you can see, there is quite a bit that goes into troubleshooting OOF Replies and it is not all straight forward.

THANKS FOR READING!
I’M AVAILABLE FOR WORK!
PLEASE CONTACT ME FOR AN APPOINTMENT!

REFERENCES:
Troubleshooting OOF Replies (Exchange Team Blog)

PowerShell – How to create a custom view for your PS Output Objects

I have been doing some training on PowerShell Scripting this week and am going to be posting a number of articles on what I have been training on. This article deal with formatting your data output from your custom script or function to be viewed the way that you want it.
Have you ever run a cmdlet where the column width is not wide enough and your data get’s truncated? Well, here is a method that you can use to make the default output of your function or script display how you want it to.

The best way to do this is by using an existing xml formatting file as a template. Run the following PowerShell commands to access those templates:

Note: your custom view xml file will need to have the “.ps1xml” extension, which indicates that it is a “Windows Powershell XML Document”.

Now to creating the custom template. Let’s say you have the following function.

Now, create a custom view using the following file as a template:

C:\WINDOWS\system32\WindowsPowerShell\v1.0\DotNetTypes.format.ps1xml

Note: You can get a list of the format type template files that already comes with PS running the following command:

Sample Output:

Mode LastWriteTime Length Name
—- ————- —— —-
-a— 10/06/2009 21:41 27338 Certificate.format.ps1xml
-a— 10/06/2009 21:41 27106 Diagnostics.Format.ps1xml
-a— 23/07/2012 19:12 144442 DotNetTypes.format.ps1xml
-a— 23/07/2012 19:12 14502 Event.Format.ps1xml
-a— 23/07/2012 19:12 21293 FileSystem.format.ps1xml
-a— 23/07/2012 19:12 287938 Help.format.ps1xml
-a— 23/07/2012 19:12 97880 HelpV3.format.ps1xml
-a— 23/07/2012 19:12 101824 PowerShellCore.format.ps1xml
-a— 10/06/2009 21:41 18612 PowerShellTrace.format.ps1xml
-a— 23/07/2012 19:12 13659 Registry.format.ps1xml
-a— 23/07/2012 19:12 17731 WSMan.Format.ps1xml

Remember: A custom view must always end with “.format.ps1xml”

We will use DotNetTypes.format.ps1xml as a template. Using this file, create a file called “hrmctools.formatps1xml”. That file will contain the following information modified from the template:

NOTE: In PS, the content of xml files are always CASE SENSITIVE!!!

Once you have created your own custom view you then need to tell PowerShell to apply the formatting by using the cmdlet Update-FormatData:

Once you run the above command, this custom format you created should now be loaded into memory. You can verify this by using the Get-FormatData cmdlet:

If you have not done so in your function or script, you can attach your custom view to your function or script using the “insert” method:

$MyObject.PSObject.TypeNames.Insert(0,’hrmctoolcustomformat’)

Note*: This code has already been inserted into our function listed in this example.
NOTE**:The first parameter “0” is something that you always type in. You can now confirm that the object has successfully been attached to the custom view, by typing in PowerShell:

Also if you output your object, you should now notice that its appearance should have now changed to those that you defined in the custom view.

A “Typename” is essentially a name that you give to your object. It tells PowerShell the type of object that it is. Know that it is possible for a number of commands to have outputting objects of the same type (the same typename value). This could affect other cmdlets and functions in your script, so be sure to debug if necessary.

HAPPY SCRIPTING!
MORE TO COME!!

REFERENCES:
Creating Custom Format Views

Exchange Server Security Update KB4540123 fails with 0x80070643

PLEASE READ THE ENTIRE POST

I had a failure on one of my two Exchange 2019 CU4 servers when installing the Security Update for them:

Exchange Server Security Update KB4540123 fails with 0x80070643

I could not restart the install as it would fail when getting to the services stoppage part of the installation. I saw this error in the ServiceControl.log file in the C:\ExchangeSetupLogs Directory

I had searched around based on the error code 0x80070643 and found these answers that some had used to get the installation to work.

LINK HERE

I downloaded the .msp file to install manually and read the answers on the web-page. It was said that there was an issue with the ServiceControl.ps1 file that is in the Exchange Server BIN directory when running Patch with the Verbose logging enabled:

I kept reading and digging into the fix for the file. It was said to modify a number of lines in the ServiceControl.ps1 script:

Once I made those changes. I renamed the original ServiceContol.ps1 to a .old file and saved this modified file to the BIN directory. I was then able to successfully run the Security Update.

BUT WHY DID THIS SERVER FAIL AND NOT THE OTHER?!?

Both are CU4, but the server that failed had been upgraded from CU1 where the one that did NOT fail was a clean installation of CU4. So, I checked the ServiceConrol.ps1 file on both servers:

Older Exchange Install
Date of ServiceControl.ps1 is 1/1/2020
CU4 Clean Installation
Date of ServiceControl.ps1 is 2/3/2020

I have not tested this, but maybe if I had copied the ServiceControl.ps1 file from the CU4 clean installation to the original install, the script might have worked since the creation dates are different and I have no other version information to go on. I will verify this though. For now, the changes to the script allowed me to successfully install the Security Update Successfully.

NOTE: All the Exchange and IIS Services were in a Startup Mode: Disabled state and I had to reset them ALL to Automatic. Once that was completed and the server rebooted, Exchange was returned to normal state. ****Also, just to be safe, I ran a great script that assures the server is out of maintenance mode. You can get the script HERE. You can also get the script to put the server into maintenance mode. You can get the script HERE. These scripts will work on Exchange 2013 and above servers.

UPDATE / 3/12/2020 2:39 AM EST

Something was still wrong with the server. ActiveSync and PAM started breaking. I started having all sorts of authentication problems. I decided to restore the server from backup from Tuesday. I forgot to backup the ServiceControl.ps1 file that modified, but I moved the one from the successful installation to the restored server and am currently running the update. I will see if it works and send the information to you.

UPDATE / 3/12/2020 3:45 AM EST

The restore worked great and placing the ServiceControl.ps1 file in the BIN directory on the prior failed Exchange Server did allow for the installation to complete successfully. I have tested ActiveSync and Authentication which is now functioning properly. Hooray!

SEND ME YOUR IDEAS/ FOR POSTS!
HAPPY TROUBLESHOOTING!

REFERENCES:
Exchange Server Security Update fails with 0x80070643
Exchange Maintenance Mode Script (Start)
Exchange Maintenance Mode Script (Stop)
Exchange Server 2019 CU4 Security Update

Adding Windows Capability to Server Core to add features needed for Application Compatibility

I’ve been working on installing Windows Server 2019 Core into my network to be able to look at new features for Windows Administration and learning how Server Core works. I was able to install a virtual machine with Server Core and get it activated. I then wanted to place my custom PowerShell script for loading PowerShell into the Server Core Environment.

So, I added the Server Core Server to the Windows Admin Center and copied my custom scripts for PowerShell into the proper directory:

Windows Admin Center
Windows Admin Center

I then logged on remotely to the server and started PowerShell. When I did that, I got this error with the script load:

Error that IE First Run has not been completed
Error that IE First Run has not been completed

At first, I tried using the -UseBasicParsing as a switch to see if that would repair the issue in the script. It did not because, IE is not installed by default on the default installation of Server Core. That is so there is less of a footprint that can be attacked by a hacker. I needed this installed though so that the Invoke-WebRequest cmdlet would load my script parameters properly.

I started looking for answers to how to install IE onto the Server Core box and found the following article. I had to run the Add-WindowsCapability cmdlet on the server to install the optional components. When I did, I received an error:

Error when adding the Windows Capability
Error when adding the Windows Capability

So I found out that there is a block that WSUS does keeping the cmdlet from going to the online source to download the software package and producing this error. After researching, I found this article. I setup a Group Policy to make sure this setting is propagated to my Server Core machine. I also setup in the same policy the ability to turn off the First-Run for IE so that you do not get that message and have to open IE to “set it up”

Group Policy Setting with Path to Templates Specified
Group Policy Setting with Path to Templates Specified

I then ran a gpupdate /force on the Server and was able to download the components for IE and App Compatibility.

Successful Installation of Windows Capability
Successful Installation of Windows Capability

I then rebooted the server and now my PowerShell loads successfully:

Successful PowerShell Load
Successful PowerShell Load

I learned a few different new things here and was able to get Server Core working more the way that I like it. I will keep posting updates when I run into issues with this type of installation. I would definitely give the Windows Admin Center a try as it has more robust features than Server Manager has, especially for Server 2019 and Server Core.

CONQUER THE UNCOMFORTABLE TO GROW!
POSITIVE ATTITUDE ABIDES!

REFERENCES:
RSAT Tools Installation Error 0x800f0954 – Windows 10 1809
Server Core App Compatibility Feature on Demand (FOD)
“Set Up Internet Explorer 11” Bypass with GPO or Registry

Set-SendConnector cmdlet does not function correctly when updating a Send Connector on an Edge Server in an Exchange Hybrid Deployment

I have run into this issue at a number of my customers that utilize an Exchange Edge Server in their Hybrid Deployment. They’ll need to modify their send connectors for their forced TLS communication with their partners or own mailboxes in Office365. Whenever they want to modify the send connector and save the changes, they get the following error messages:

Symptoms

“PowerShell failed to invoke ‘Set-SendConnector’: Error 0x5 (Access is denied) from cli_GetCertificate”

or

“Error 0x6ba (the RPC server is unavailable) from cli_GetCertificate”

This issue occurs after you install the Cumulative Update 14 for Exchange Server 2016Cumulative Update 13 for Exchange Server 2016, or Cumulative Update 23 for Exchange Server 2013.

Cause

This issue occurs because the TLS certificate check (in case the TlsCertificateName attribute is populated on the send connector) doesn’t work against the Edge servers as the RPC communication is blocked against the Edge servers.

Workaround

Now the current workaround for this has been to delete the Edge Send Connector and recreate the connector from scratch via PowerShell with all the settings and changes entered. This is not a viable solution especially if your communications with your partners change constantly and changes are made to the secure communications channel between you and them.

Resolution

To fix this issue, install one of the following updates:

For Exchange Server 2019, install the Cumulative Update 4 for Exchange Server 2019 or a later cumulative update for Exchange Server 2019.

For Exchange Server 2016, install the Cumulative Update 15 for Exchange Server 2016 or a later cumulative update for Exchange Server 2016.

For Exchange Server 2013, there is no fix at this time. My personal recommendation is to plan an upgrade to Exchange 2019.

KEEP POSITIVLY MOVING FORWARD!

REFERENCES
Set-SendConnector doesn’t work for Exchange Server in hybrid scenarios with Edge Server installed

RBAC Role Assignments NOT installed during Exchange Directory Preparation

I had a very interesting installation issue recently when installing Exchange 2019 into a new environment. We ran through all the Exchange Preparation for the root and child domains in the forest as described HERE. The results of those installation procedures showed SUCCESS, but when we started installing Exchange, we ran into issues with the System Mailboxes not being available to complete the Mailbox Role part of the installation. Most of the articles that I found said to re-run the Domain Prep (/preparealldomains) and the AD Prep (/pad). So we did, and managed to get the first server installed somehow.

The reason I said somehow is because when we tried to logon to the EAC, we would get a 400 Bad Request Error and could not logon to the console. Next, we tried PowerShell and was able to load PowerShell, but I noticed that only ~100 cmdlets loaded. I thought that maybe we had to re-create the account mailbox to get it working properly. Problem was, one of the cmdlets that would not load was Disable-Mailbox along with others like Enable-Mailbox and New-Mailbox. It was as if the admin account we were using had no rights to administer Exchange in any way.

Next, we opened the mailbox in OWA. The mailbox came up okay, so I told the admin to change the URL to /ecp to try and get into the admin center. What happened was that the normal user control panel opened instead, showing again that the account did not have permissions.

We checked replication to the child domain and made sure there were not any apparent AD issues present. There were none. I next started reviewing how Exchange uses RBAC (Role Based Access Control) Groups and Role Assignments to grant users access to Exchange Admin Functionality. I read the following article located HERE.

Something told me to go and check the Schema again, so I went to ADSIEdit > Configuration Container > Services > Microsoft Exchange > (Organization Name) > RBAC > Role Assignments

I looked at the list of role assignments in the window as follows:

Small List of RBAC Role Assignments
RBAC Role Assignments Missing Objects

From the picture, you can see that the list is small, which in my experience is not correct. I verified this by going into my own 2019 environment and comparing the number of objects in that folder:

RBAC Assignments Object list with CORRECT Objects Listed

If you notice the list is MUCH longer and has many more objects listed in the container. So, how did Exchange Setup miss this during preparation? That I will find out later, but first I have to remediate this problem.

CAUSE:

If the RBAC roles assignments are not installed to allow an account to have administrative privileges in Exchange, then you cannot administrate Exchange to even make the necessary changes! Especially so if you’ve only installed ONE server in the environment!

REMEDIATION:

Manually repair the installation by running the script that creates these Objects in the Schema during setup.

******DISCLAIMER: Running the following commands in these instructions, running ADSIEdit, and/or making changes to your Schema and Exchange Installation outside the normal setup process is NOT recommended! Microsoft, LDLNET LLC, nor I (Lance Lingerfelt) are responsible for any issues or errors that may arise from using these instructions, period!******

That said, preform the following to regenerate the objects in the Schema:

1) Open Windows PowerShell (not the Exchange Management Shell) on the server that you installed Exchange Server on with the same account you used to install Exchange.

a. If you have UAC enabled, right click Windows PowerShell and click Run as administrator.

2) Run Start-Transcript c:\RBAC.txt and press Enter

a. This will start logging all commands and output you type to a text file.

3) Run Add-PSSnapin *setup and press Enter

a. This adds the setup snap-in which contains the setup cmdlets used by Exchange during install. You may see errors about loading a format data file. You can ignore those errors.
NOTE: DO NOT run any other cmdlets in this snap-in. Doing so could irreparably damage your Exchange installation.

4) Run Install-CannedRbacRoleAssignments -InvocationMode Install -Verbose and press Enter.

a. This cmdlet should create the required role assignments between the role groups and roles that should have been created during setup.

b. Be sure you run with the Verbose switch so we can capture what the cmdlet does.

5) Run Remove-PSSnapin *setup and press Enter

6) Run $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://servername/PowerShell/ -Authentication Kerberos and press Enter

a. Be sure to replace SERVERNAME with the FQDN of your server.

7) Run Import-PSSession $Session and press Enter

a. You should notice that the normal number of cmdlets load (~700)

8) Run Get-ManagementRoleAssignment and press Enter. If you are able to run the cmdlet, then the remedation worked.

9) Run Stop-Transcript and press Enter

The final check is to return to ADSIEdit and check the container and see if all the objects are there. We also were now able to get into EAC as well as saw that the Arbitration mailboxes were populating along with the Health Mailboxes as needed per the installation.
It was very neat to see how running the Add-PSSnapIn cmdlet opened all the scripts from the Exchange Setup and allowed me to manually fix the installation problem by running the cmdlet script that need to perform that task that setup missed or refused to run.

POST MORTEM REVIEW

I am going to look over the installation logs and see where the installation failed and try to find out why it did not run on the subsequent re-installations of the AD Prep and Domain Prep. I will post those finding in this article when I have that available.
Thanks again to my Microsoft and Trimax teammates for your assistance with this. It has helped the customer in more ways than one!

HAPPY TROUBLESHOOTING!
POSITIVE ATTITUDE YIELDS POSITIVE RESULTS!

Exchange Server Client Access URL Configuration Script

In my career, I have to be able to be efficient as most of my projects are on a time crunch schedule. Being able to quickly configure Exchange when setting up a server environment is crucial to the success of the project.

While still honing my skills in PowerShell, I was attempting to create my own script to help configure all of the Virtual Directories in one shot rather than go to each setting and configure them manually. It did not go very well, so as I do, I research and find great professionals that do great work in scripting so that I may learn from them.

In doing so, I found Paul Cunningham’s script that performs this. I took the following script and modified it to add the PowerShell Virtual Directory to it as I like to configure that as well.

***YOU CAN REM THE LINES OUT SHOULD YOU NOT WANT TO CONFIGURE THAT DIRECTORY***

Here is my version of the script:

NOTES:

  • PowerShell script to configure the Client Access server URLs for Microsoft Exchange Server 2013/2016. All Client Access server URLs will be set to the same namespace.
  • If you are using separate namespaces for each CAS service this script will not handle that.
  • The script sets Outlook Anywhere to use NTLM with SSL required by default.
  • If you have different auth requirements for Outlook Anywhere use the optional parameters to set those.
  • The script sets PowerShell to use Basic with SSL required by default.
  • If you have different authentication requirements for PowerShell use the optional parameters to set those.
  • PowerShell was added to the settings. Please be sure to REM those lines of code should you NOT want to configure the PowerShell Virtual Directory.

USAGE:

HAPPY SCRIPTING!
POSITIVE ENERGY!
PLEASE COMMENT!

REFERENCES:
Exchange Server Client Access URL Configuration Script
PowerShell Script to Configure Exchange Server Client Access URLs

How to Stop and Start All SharePoint 2013 Farm Services using PowerShell

How to Stop and Start All SharePoint 2013 Farm Services using PowerShell? 

Prior to SharePoint patching, its a best practice to Stop all SharePoint 2013 and its related services and then start once patching is completed. If you don’t do this, your service pack or patch installation will take longer than its expected.

So what are all the services to be stopped?

SharePoint 2013 Search Service (OSearch15 – OSearch16 in SharePoint 2016)

SharePoint 2013 Timer Job (SPTimerV4) • SharePoint 2013 Administration (SPAdminV4)

SharePoint 2013 Tracing (SPTraceV4)

SharePoint 2013 VSS Writer (SPWriterV4)

SharePoint 2013 User Code Host (SPUserCodeV4)

• SharePoint Search Host Controller (SPSearchHostController)

• Forefront Sync Service (FIMSynchronizationService)

Forefront Service (FIMService)

World Wide Web Publishing Service (W3SVC)

Internet Information Services (IIS)

Don’t forget to do it in all your SharePoint Servers of the farm!

Lets use PowerShell to stop and start all SharePoint services:

Stop all SharePoint 2013 Services, Lets use PowerShell to stop and start all SharePoint services:

Start all SharePoint 2013 Services: After the patching, Use the below script to start all SharePoint services. 

Completely Stop or Start SharePoint Farm Services on All Servers: Lets put everything together and make a reusable PowerShell function, which stops or starts all SharePoint related services in all servers of the farm.

Installing an ‘IP-less’ Exchange Server 2019 Database Availability Group

Yesterday, I posted on how Exchange now uses the Resilient File System (ReFS) to optimize and protect Exchange critical files. Another layer of protection is using a database availability group (DAG) for redundancy and is a necessary factor when designing an Exchange Enterprise Environment.
In this example, I will walk you through the installation of an Exchange Server 2019 DAG as I configured in my environment. This DAG will contain two Exchange Servers in the same site with a third Windows Server 2019 server being the File Share Witness (FSW).

Two Server Exchange DAG Configuration

For my configuration, I configured two identical Windows Server 2019 VMs (same procs, RAM, vhdx drives, partitions, etc…). I configured the Exchange Data Volume using ReFS and mounted them to the same folder on the C: Drive on each server. This is very important for replication to take place successfully when the databases are added to the DAG.


I next went to the Admin server where the FSW would be hosted and added the Exchange Trusted Subsystem Account to the local Administrators group on that server:

IMPORTANT!
Add the Exchange Trusted Subsystem Account to the Local Administrators Group on the FSW.

NOTE: The reason that this is an ‘IP-less’ DAG is that I’m creating a DAG with no cluster administrative access point (CAAP). The DAG has no IP address of its own, and no computer object in Active Directory. The main implication of this is that backup software that relies on the CAAP or backup operations won’t work. This option of an ‘IP-less’ DAG was first introduced in Exchange Server 2013 SP1/CU4, so by now any decent backup products should support this configuration. But you should always verify this with your backup vendor of choice. Also be aware that this is only supported for DAGs that are running on Windows Server 2012 R2 (or later).

Next, we create the DAG from Exchange PowerShell using the New-DatabaseAvailabilityGroup cmdlet. Now remember that since you are using the ReFS system for your database volumes, you will need to specify the -FileSystem parameter within the cmdlet to assure proper setup and replication of the data files.

Next, we add the Exchange Servers that hold the databases that will be replicated within the DAG:

The DAG will now show the two servers as Operational Member Servers:

The FSW Directory was created on the admin01 server when the DAG was created. We can verify that with the following cmdlet:

Next, we add the databases that we want replicated to the DAG as replicated databases. I want all my Databases on EX01 to replicate to EX02 and vice versa for the EX02 Databases. I want the activation preference to remain on the server that the databases were originally created on so I will use the -ActivationPreference parameter to accomplish that. I will go into more detail on Activation Preference in another post.

Now we verify that the Database Copies are healthy on each replication member using the Get-MailboxDatabaseCopyStatus cmdlet. You will see a Healthy Status on the replicated copies:

POSITIVE ENERGY!
KILL NARCISSISM!
HAPPY TROUBLESHOOTING!

REFERENCES:
Installing an Exchange Server 2016 Database Availability Group

Hyper-V General Access Denied error when trying to load a Virtual Hard Drive and start a VM

I was working on setting up a VM for my server farm and mis-configured one of the vhdx drives. I ended up having to delete that drive and recreate it in Hyper-V manager. When I did though, I received an error stating that I could not start the virtual machine:

An error occurred while attempting to start the selected virtual machine(s).
‘VMName’ failed to start. (Virtual machine ID ‘SomeID’)
‘VMName’ Microsoft Emulated IDE Controller (Instance ID ‘SomeID’): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)
‘VMName’: IDE/ATAPI Account does not have sufficient privilege to open attachment ‘C:\Users\Public\Documents\Hyper-V\Virtual hard disks\DiskName.vhdx’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)
‘VMName’:  Account does not have sufficient privilege to open attachment ‘V:\Hyper-V\Virtual hard disks\DiskName.vhdx’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID ‘SomeID’)

Causes

Each virtual machine is started using a virtual machine account. The virtual machine account needs read and write access to the .vhd/.vhdx file, but if the file has just been copied from somewhere then it most likely lacks the necessary file permissions.
That happened in my case because I had just created the vhdx drive and did not create it from the VM itself. I just attached it to the VM. So, when I booted the VM, it gave the error.

Remediation

There are a few ways that you could remediate the issue. The simplest way, if it is a new VM, is to remove the drive in the VM settings and then re-create it from scratch. That is what fixed it for me.
Another way is to add the VM GUID to the permissions so that it can access the vhdx file properly:

  • If you don’t already have the Hyper-V Manager error dialog open (“An error occurred while attempting to start the selected virtual machine(s) …”) then try to start the virtual machine now. You need the error open.
  • Click “See details”. This will show additional details, and will look something like:

‘PC-Name’ failed to start. (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD) ‘PC-Name’ Microsoft Emulated IDE Controller (Instance ID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX): Failed to Power on with Error ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
‘PC-Name’: IDE/ATAPI Account does not have sufficient privilege to open attachment ‘E:\Hyper-V\PC-Name\Virtual Hard Disks\MyVHD.vhdx’.
Error: ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
‘PC-Name’: Hyper-V Virtual Machine Management service Account does not have sufficient privolege to open attachment ‘E:\Hyper-V\PC-Name\Virtual Hard Disks\MyVHD.vhdx’.
Error: ‘General access denied error’ (0x80070005). (Virtual machine ID B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD)
Where PC-Name will be the name of your virtual PC. The long sequence of letters and numbers (in my case above “B9C4F7D4-0009-4BE2-90FB-9D60B1A06BDD”) is the Virtual Machine ID. This number is significant and you need it to fix the problem.

  • On the host server open an elevated command prompt.
  • Enter the following:

You will need to substitute the path to the vhd/vhdx file – you can obtain this from the original error message, and the Virtual-Machine-ID that you obtained from the “See details” part of the error.

So the line for me was:

NOTE: If you get the message “Failed processing 1 files” then check the virtual machine ID.

  • Now try to start the virtual machine. The error should no longer be present.

There is also a PowerShell Gallery script that is supposed to remediate this issue:

http://www.ntsystems.it/page/PS-Restore-VMPermissionps1.aspx

I haven’t tried it but it looks as it would work. Please review and leave a comment should you have issues with the script.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!
POSITIVE ENERGY!

REFERENCES:
Resolved: Hyper-V General access denied error when trying to load a Virtual Hard Drive
Restore-VMPermission
Virtual machine fails to start with General access denied error / Account does not have sufficient privilege to open attachment

Importing User Photos to Office 365 in bulk for your company.

In a previous post, I showed how you could update one user’s photo for their Outlook and AD profiles via PowerShell. In this post, we will explore how to do this for your entire organization via PowerShell to Office365.

NOTE: I have not tested the scripts as I do not have enough mailboxes in my O365 tenant along with not using a ‘.’ in my alias. If the scripts are incorrect, please inform me with the correction and I will update accordingly.

Please make sure that your photos are reviewed before posting, and try to keep the file size of the photos to a minimum. In Office 365, there exists a limitation for the user photo not to be more than 10 KB in size, but I will show you how to get around that limitation.

Having a user photo for each of your users is very beneficial as it personalizes each account to a face in the company. The user photos can be viewed in below locations:

  • Outlook Web Access
  • Contact Card
  • Thumbnail in emails
  • Outlook Client
  • Yammer
  • Lync Client
  • SharePoint (People Search / Newsfeed)

Steps to take:

  1. Remove the 10KB photo size limitation in Exchange Online
  2. Prepare a folder with all users photos
  3. Update the profile photos via a PowerShell cmdlet.

Connect to Exchange Online with the RPS Proxy Method to remove the 10K size limitation

NOTE: In the PowerShell cmdlet above, we connected using a different proxy method. This was to overwrite the limitation of uploading the images with size more than 10KB. Using the different proxy method (/?proxyMethod=RPS ) to connect to Office 365 in the above cmdlet accomplishes this.

Prepare a folder locally and place all the photos in that folder

Create a folder named C:/UserPics and make the filename of each photo be the username of that particular user. (i.e. llingerfelt.png)
The below script should be able account for aliases that have a ‘.’ in the id as well. (i.e. lance.lingerfelt)

NOTE: From my research, there is no set photo type that is required for the photo. My suggestion would be to keep the photos .png for size constraints while maintaining picture clarity.

Update the profile pictures via PowerShell

Create the following script and name it Photos-Update.ps1

Run Photos-Update.ps1 and the script should upload the photos to Office 365 and apply each photo to the corresponding user.

NOTE: If you’re still having some issues with the alias having a ‘.’ in the name, you can also configure the Photos-Update.ps1 script in this manner to get that working properly:

HAPPY SCRIPTING!
PLEASE COMMENT!

store.ldlnet.net
LDLNET LLC! Your Source for Pofessional IT Services!
www.servermonkey.com
Contact ServerMonkey.com for your IT Hardware Needs!

REFERENCES:
How to import Office365 User photos over 10KB & without CSV in bulk

Set the profile pic for a single Exchange user via PowerShell

I wanted to update my picture within my Outlook profile and AD account really quickly without having to go through OWA to do so. I found this cmdlet that will allow for that picture to be changed very quickly via Exchange PowerShell.

NOTE: This can be done with On-Premises Exchange and Exchange Online PowerShell

Old picture within my account

First, download the picture you want to use to the computer that you want to run the cmdlet from. Also, make sure the picture is cropped and centered prior to running the cmdlet. I saved the pic to C:\temp for my scenario. The best format to use would be jpg. I named the file User1_Profile.jpg

Next, open Exchange PowerShell on the computer you saved the pic to and run the following cmdlet to change the photo:

Once completed, the Outlook client should be closed and reopen so that the new picture is visible in the profile.

Picture change completed

I will post how to perform this for multiple users for Exchange and Office365 in a later post.

REFERENCES:
Set User Photo with Exchange PowerShell

Purging Soft Deleted mailboxes from Exchange Server

If you’re a seasoned administrator, you have knowledge that in Exchange, the database settings will allow you to set the deleted mailbox retention. The default is 30 days, but sometimes you need to purge all those deleted mailboxes to do some ‘spring cleaning’ as it were. Note that doing these cmdlets does not change the ‘Whitespace’ of the database or the size. In my case, I had to purge everything of a toxic individual that was tainting my network much to my disappointment and did the following to complete that task.

The following cmdlet will seek all Soft Deleted mailboxes within the database you select and manually purge them from Exchange.

Now, should you only want to remove one mailbox, you will need to get the GUID of that Soft Deleted mailbox first so that you can enter it for the identity parameter.

You can also preform a similar task for a disabled mailbox:

You can perform the task on all disabled mailboxes for that database as well:

NOTE: I would be very careful when performing either of these cmdlets as they will completely purge the mailboxes from the schema. If these cmdlets assist you with your ‘spring cleaning’, I will have been happy to assist.

HAPPY PURGING!
PLEASE COMMENT!
IGNORANCE IS NOT BLISS!

References:
Purging Deleted Mailboxes on Exchange 2013

Server Monkey
LDLNET LLC’s Preferred Server Equipment Hardware Vendor

Installation of Exchange Server 2019 on Windows Server 2019

I have realized recently that I am an Exchange Messaging Professional, but yet, I have not posted the methodology of how I install an Exchange Server Mailbox Role. So here it is!

Install Windows Server 2019

Exchange Server 2019 requires Windows Server 2019 to run. For my environment, I haven’t necessarily need to follow all the enterprise level design aspects of database numbers to mailbox size ratios, number of servers, front/back end configurations, DAG Implementation, etc… If you want or need to delve into that realm, you can go here. I have need for a single server with only a few databases for a small number of mailboxes, so I am approaching it from that standpoint.

So first, in Hyper-V, I configured my VM with the following specifications:

Processors: 2 procs with 2 cores each – 4 Virtual Processors Total
RAM: 32GB with dynamic memory enabled optional
Drives: 2 .vhdx drives of 120GB each (OS / Exchange Data)
CD: Windows 2019 ISO
Default Settings for the rest of the VM Settings

Next I installed Windows Server 2019 Datacenter with the GUI! You can install it on Server Core if you wish. That information can be found in this link.

I ran through the setup of Windows and installed the OS on my first vhdx drive. I booted up, set the local admin password, and logged in. Once in Windows, I went to the Local Server Settings in Server Manager and configured the following settings:

Set the Date, Time, and Time Zone. (Once in the Domain, this would sync through Group Policy)
Set IE ESC to allow Administrators to have full IE access.
Set Remote Desktop Settings to gain RDP access. (This would be locked down with Group Policy as well once on the Domain)
Set the IP Settings to Static Settings. (DNS Servers, Gateway, WINS, etc…)
Join the server to the Active Directory Domain.
Reboot the VM Server.
Logon to your Domain.
Configure Windows Update Settings. (I have WSUS through Group Policy, this was configured automatically upon reboot)
Download and install all Windows Updates for the server. Then Reboot.
Open Disk Management and configure the secondary vhdx drive to be your Exchange Data Drive.
I configured the drive to be a mounted folder ‘C:\Exchange\Data’ rather than another drive letter as that seems to be the more accepted form of installation for the data drive these days. That is based on the multiple configurations that I have seen for Exchange through experience in Enterprise environments. Again, to each is own and depending on you design specifications, you might want to do that differently.

Next, we need to install the prerequisites for the Exchange Mailbox Server. I have always used practical365.com to get the PowerShell script to install the prerequisites, but couldn’t find the article this time. Great site though! Instead, I got the information and ran the following from an elevated PowerShell Session locally on the server:

As part of the prerequisites you will need to install the following packages onto the server as well:

UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013

Once completed, you can begin the install of Exchange. If this is your first Exchange 2019 Server in your Organization, then you will need to run the following to update the Forest, Schema, and Domain so that Exchange will install properly:

NOTE: If you run into Prerequisite issues with the installation due to a “pending reboot”, check out my blog post for information on remediation of that issue.

Now that the environment is prepared for Exchange, you can actually begin the installation. I wanted to make my default database and logs folder to be on the Exchange Data volume that I created, so I included those settings in the setup command. Please look at the reference to the setup.exe switches for more information on that. Here is the command:

Setup should go through the installation via the PowerShell window and complete successfully. Reboot the Exchange Server, then you can then logon to the Exchange Admin Center and begin the process of configuration of how you need to integrate the Mailbox Server into your Server Farm. That configuration is for a later post.

PLEASE CHECK BACK FOR UPDATES!
PLEASE COMMENT!

References:
UCMA Runtime Install
Visual C++ Redistributable Packages for Visual Studio 2013
Install Exchange Server 2019 on Windows Server 2019 Core
Exchange Server Design Planning
Use unattended mode in Exchange Setup
Practical365 on Exchange 2019

Removing Hidden Devices in Device Manager

As you may have knowledge of, if you are reading my blog. I am currently migrating off of VMWare to Hyper-V. Now, as I convert my machines to Hyper-V, it uses a totally different driver for the Network Card. I am having to rebuild the NIC settings within windows to setup the NIC for the Hyper-V VM to get the machines on the network properly again. The VMWare NIC disables and hides the NIC from the VMWare driver in Device Manager.

What this does is make Windows think it has two active network cards, even though one is disabled and removed/hidden in device manager. So, to clean things within Windows, I have to perform the following procedure to remove the hidden device:

Open PowerShell as Administrator
Next, type the following cmdlet and press Enter:

Next, open Device Manager from the PowerShell Session:

When the Device Manager GUI opens, click the View menu
Click 
Show Hidden Devices
Go to the Device that is hidden, in my case the Network Adapter
Right-Click the Device and select Uninstall

Close the Device Manager GUI and PowerShell session

This cleaned the old hardware drivers off the system and allowed the current Hyper-V NIC to be the only one installed.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!

How to transfer FSMO Roles using PowerShell

A rare weekend post for me! HA! I am currently migrating my server environment from VMWare 6.7 to Server 2019 Hyper-V. I have a separate standalone box that I use for my VM backups and as a tertiary DC. Since I had to shut down my VMs in order to convert them, I needed to quickly move my FSMO roles from the DC Virtual Machine to the Standalone box so things would stay running.

I found this great article on how to do that quickly through PowerShell since it is a pain to go into ADUC, ADDT, and setup an MMC for the Schema snap-in.

When you create a domain, all FSMO roles assigned to the first domain controller in the forest by default. You can transfer FSMO roles from one DC to another both the Active Directory graphics snap-ins and the PowerShell command line. Moving FSMO roles using AD PowerShell has the following benefits:

  • You do not need to connect with a MMC snap-ins to the future role owner;
  • Transferring or seizing FSMO roles does not require a connection to the current or future role owner. You can run AD-PowerShell module cmdlets on a Windows Client or Server running RSAT Tools;
  • To seize the FSMO role (if the current owner is not available), it suffices to use an additional parameter -force.

Import the Active Directory Module Into PowerShell:

To get the current forest level FSMO role owners (Domain Naming Master and Schema Master roles) you can use the following PowerShell cmdlet:

To view domain-wide FSMO roles (Infrastructure Master, PDC Emulator and Relative Identifier Master roles):

Transfer FSMO Roles using PowerShell

To transfer FSMO roles between Active Directory domain controllers, we use the PowerShell cmdlet:
Move-ADDirectoryServerOperationMasterRole

To use the Move-ADDirectoryServerOperationMasterRole cmdlet, you must meet the following requirements:

  • There must be at least one DC with a version of Windows Server 2008 R2 or higher
  • PowerShell version 3.0 or newer
  • Active Directory module (2.0  or newer)

NOTE: Unlike the Ntdsutil.exe utility, the Move-ADDirectoryServerOperationMasteRole cmdlet can be performed from any domain computer to migrate the Operations Master roles if you have the appropriate rights (Domain admins and Enterprise Admins).

Import the AD Module:

I needed to move all the roles from one server to the other, so, I ran the following to do so:

NOTE: To simplify the command, you can replace the names of roles with numbers from 0 to 4. The correspondence of names and numbers is given in the table:

PDCEmulator0
RIDMaster1
InfrastructureMaster2
SchemaMaster3
DomainNamingMaster4

So, by having knowledge of these numbers, you can simplify your cmdlet:

NOTE: In the event that the current owner of one or all of the FSMO roles fails, the forced transfer of FSMO roles is performed by the same command, but with the -Force option. Also, after the FSMO roles have been seized, the domain controller from which the roles was seized should never be connected to the domain. You will need to preform a metadata cleanup of the Schema before even thinking about putting that failed server back into production.

Once completed, I ran the previous cmdlets of Get-ADForest and Get-ADDomain to verify that the FSMO roles moved to the destination server.

As of now, my conversion to Hyper-V is going smoothly, although it takes quite a bit of time to convert the hard disks. Thanks again!

HAPPY TROUBLESHOOTING! KEEP SCRIPTING!
PLEASE COMMENT!

Reference:
How To Transfer FSMO Roles Using PowerShell

Generate Automated E-Mail From Performance Counter Alert

Recently, I was trying to load this IT Blog and noticed that the page took a very long time to load properly. So, first thing I did was logon to my Web Server to check things and saw that the Performance was very slow, almost to a halt. When I was able to pull up Server Manager, I saw the following:

The Performance Alert was pegged at 99+ alerts. I went into the alerts for the performance and saw the following:

My CPU Usage was pegged at 100% for many hours. As I was having issues with the server and could not dig any deeper to find what was running to cause the error other the the w3wp process, I decided to reboot the server.
Rebooting did resolve the issue at hand, but I needed a better way to alert myself when the processor pegged at 95% or more over at least a steady two minute period. That way I could keep the websites going and restart the server if necessary due to an issue with a process pegging the CPU.

I don’t have SCCM or any robust alerting software like we have a my employer, but not to worry, Windows Server has a built in method to generate a task using Task Scheduler when the Performance Counter is above threshold for the time period that you set.

Step 1 – Create your PowerShell Script

The first thing that you need to do is write a Powershell script that when run can send an email. While researching this I discovered many ways to accomplish this task, feel free though to experiment and use what is right for your environment.

Here is an example of a Powershell script that when used in conjunction with Task Scheduler and Perfmon can send an email alert automatically when any user defined performance counter threshold condition is met. In my environment I set this to C:\PowerShell\AlertEMail.ps1

If you notice, this Powershell script takes four arguments and assigns them to variables used in the output. It also saves the computer name to a variable which is also used as part of the output. By doing this the script can be used to send an email on any Perfmon Alerting Counter and on any server without additional customization of the script.

Step 2 – Setup a Scheduled Task To Be Run

In Task Scheduler we are going to Create a new Task as show in the following screen shots.

Create Task

Give the Task a name, you will need to remember it for the next step. Make sure the task is run with an administrative account and is run whether the user is logged on or not.

There are no Triggers to setup here. This Task will actually be triggered through the Perfmon Counter Alert we will set up in Step 3.

On the Action tab you want to define a new action. The action will be to Start a Program and use the following inputs, please adjust for your specific environment.

Program/Script: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Add Arguments: -File C:\PowerShell\AlertEMail.ps1 $(Arg0)

Next, set your Conditions and Settings Tabs to the following settings as I did for this purpose. If need be, you might need to set them differently, but these settings should work just fine.

Step 3 – Create the Alert Trigger in Performance Monitor

Open Performance Monitor and Create a new Data Collector Set under Data Collector Sets > User Defined

Name the Alert (This one is for the Processor Usage)
Assign it as a Performance Counter Alert

I needed to monitor the Total % Processor Time for all processors on the server combined, so I set the counter to monitor from the available counters.

I then set the threshold to Alert when the processor is over 95% with Processor Time. I also let it run as the System account, and then saved the settings:

Once you have created the Data Collector Set go into the Properties of it and make sure the Alerting threshold and Sample Interval is set properly for each Performance Counter. Keep in mind, if you sample every 10 seconds then you should expect to receive an email every 10 seconds as long as the performance counter exceeds the threshold you set. I set my Sample Interval to 2 minutes so that if the percentage stays above 95% over that long of a time interval, I should take a look at the server and see what process is pegging the CPU:

If you select the Log an entry in the application event log don’t expect to see any entries in the normal Application event log. It will be written to the Microsoft-Windows-Diagnosis-PLA/Operational log in the Application and Services log directory.

Finally, we have to set an Alert Task that will trigger the Scheduled Task (AlertEMail) that we created in Step 2. You see that we also pass some of the Task arguments which are used by the PowerShell script to customize the email with the exact error condition associated with the Alert:

Once the Data Collector is configured properly you will want to start it.

If you configured everything correctly you should start receiving emails any time that alert threshold is surpassed. I also manually ran the script once from PowerShell with no Argument Data just to make sure the script would generate the email to my E-Mail Address.

Step 4 – Set the Data Collector Set to run upon server startup.

You now have one more step. Whenever you reboot a server the Perfmon Counter Alert will not start automatically. In order to survive a reboot you must run the following at a command prompt.

Note: “Processor Usage Alert” referenced in the script below is the name of my user defined Data Collector Set.

I now have a simple way to monitor my Web Server should this event happen again with the processor being pegged over a two minute period.

PLEASE COMMENT! I LOVE IMPROVEMENTS TO MY SCRIPTS! HAPPY TROUBLESHOOTING!

Reference:
Setup E-Mail Generation from a Performance Counter Alert

Check Windows Updates Installed via PowerShell

I had an issue last night where a server lost Secure Channel Connection to the PDC Emulator (NETLOGON Event IDs 5719 and 5783). All tests to test the secure channel via PowerShell were failing. (i.e. nltest or Test-ComputerSecureChannel cmdlets) The server essentially needed to be rebooted. I had a dumb dumb in my brain and forgot to check to see if there were any pending Windows Updates, because those need to be installed at the proper time and to a schedule. So, when I ran the following command to reboot:

The Windows Updates were installed inadvertently which could have caused even more issues if they were NOT approved or caused another failure on the server. TO NOT DO THIS IN THE FUTURE, remember to run the following command to shut off the Windows Update Service BEFORE initiating the reboot of the server:

But, the deed was done. NOW, I had to find out quickly what updates WERE installed via PowerShell so that I could alert the proper folks and give them a heads up on possible issues. Luckily, the server did NOT have any issues and the initial problem with NETLOGON was resolved. Here is the command I ran to find out the installed hotfixes filtered by today’s date:

Here was the Output:

Caption=http://support.microsoft.com/?kbid=4480960 
CSName=DC01
Description=Security Update 
FixComments= 
HotFixID=KB4480960 
InstallDate= 
InstalledBy=NT AUTHORITY\SYSTEM 
InstalledOn=2/26/2019 
Name= 
ServicePackInEffect= 
Status= 

Caption=http://support.microsoft.com/?kbid=4480965 
CSName=DC01 
Description=Security Update 
FixComments= 
HotFixID=KB4480965 
InstallDate= 
InstalledBy=NT AUTHORITY\SYSTEM 
InstalledOn=2/26/2019 
Name= 
ServicePackInEffect= 
Status=
 

Since there were no issues, I was able to resolve the incident. I did notify the account team though of the inadvertent installation so that they could revert the changes if necessary.

Remember, troubleshooting to resolution is a methodical process, and when in an enterprise environment, you MUST be aware of all factors of change process, even when the resolution is a simple reboot of the affected server.

HAPPY TROUBLESHOOTING!
I LOVE COMMENTS! THANKS FOR READING!

References:
Methods of generating installed updates via PowerShell
Check Windows Update History via PowerShell
Disable or Bypass Windows Update Installation During Reboot/Shutdown of a Server