Protected AD Groups and the problems they can cause accounts

I have run into this issue over the years with accounts being in the Domain Admins group and having issues running PowerShell cmdlets as well as not being able to connect to ActiveSync from a mobile device with the account.

These issues are due to the AdminSDHolder Template in AD and the SDProp Process that is run every 60 Minutes in AD.
This is explained in fantastic detail through the following Microsoft article: Protected Accounts & Groups In Active Directory

Here is an example of an issue that occurred in one of the environments that I was managing. A user was trying to run the following AD cmdlet in PowerShell on DC01:

The user got the following error when the cmdlet was executed:

Set-ADUser : Insufficient access rights to perform the operation
At line:1 char:1
+ Set-ADUser lancel -Server dc01.ldlnet.org -Replace @{title=”Senior O …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo: NotSpecified: (lancel:ADUser) [Set-ADUser], ADException
+ FullyQualifiedErrorId : ActiveDirectoryServer:8344,Microsoft.ActiveDirectory.Management.Commands.SetADUser

The issue was that the admin account used to run the cmdlet was in the Domain Admins group and was not inheriting permissions per the AdminSDHolder template that was applied to the account:

I checked to see that the admin account was in a protected group:

I next went to the Security Tab > Advanced Button and saw that the Enable Inheritance button was visible:

I’ve circled where to look in the window.

This verifies that the account is protected due to being in the Domain Admins group. Now, there are two workarounds for this particular error that we were experiencing.

  1. Click the Enable Inheritance button. This will cause the permissions to be inherited temporarily. When SDProp is cycled again, the account will lose any inherited permissions and will be essentially “broken” again. This is not good if you’re going to be running cmdlets regularly to modify AD Accounts.
  2. The preferred method to work around this issue is to set the -Server parameter to point to a different DC than the one you are on. So, essentially, we tell the cmdlet to execute on DC02 when running the cmdlet from DC01.

Either method will allow the cmdlet to execute successfully and modify the object. You would think that Microsoft would have noticed this issue with running an admin cmdlet for Active Directory, but they have not fixed this issue as of yet nor do i think they plan to. I would just go with workaround number two and remain sane.

Another example of this Protected Group issue comes with an account in a Protected Group that has a mailbox not being able to connect to Exchange ActiveSync when setting up their mobile device.

  • You usually get a 500 error on the device that you cannot connect.
  • You will also see event 1053 in Event Viewer alluding to not having sufficient access to create the container for the user in AD.

Read this page for more information: Exchange ActiveSync Permissions Issue with Protected Groups

So, in your endeavors admins, keep this in mind when running into these types of problems. Happy Troubleshooting!

Exchange Server HealthSets

This is a monitoring feature included with Exchange that until recently, I did not know existed, as it wasn’t really mentioned in any of my dealings with Exchange Server up until recently. The HealthSets feature monitors every aspect of a running Exchange Server and is broken down into three monitoring components:

  • Probe: used to determine if Exchange components are active.
  • Monitor: when probes signal a different state then the one stored in the patters of the monitoring engine, monitoring engine will determine whether a component or feature is unhealthy.
  • Responder: will take action if a monitor will alert the responder about an unhealthy state. Responders will take different actions depending on the type of component or feature. Actions can start with just recycling the application pool and can go to as far as restarting the server or even worse putting the server offline so it won’t accept any connections.

From what I have experienced in the past year with these HealthSets, an alert will be thrown due to a change in a service, or a restart of a service, a failed monitoring probe result, or something of the like. The healthset will become “unhealthy” in state at that time. You can run the following on a server in order to get the healthset status of that server:

If you get alerts for multiple Exchange Servers, let’s say for instance, the transport array, you can run the following cmdlets to get the status of all the Transport Servers in the array:

HealthSet PowerShell Output
HealthSet PowerShell Output

Now, a lot of times, the Unhealthy value in the HealthSet will have corrected itself as per the Responder, even though the AlertValue will remain Unhealthy. To clear the cache quickly and have the monitor probes run again for verification, perform the following restarts of services from this cmdlet in this order:

That should clear the probe results and let them run again. Now, should they again return an error, we will need to dig deeper to figure out the issue.
What you will want to do first is get the monitor definition. In this example, the Mapi.Submit.Monitor was the component that was unhealthy in the healthset. I had to run the following cmdlet to get the Monitor Definition:

Output:

auto-ns2                           : http://schemas.microsoft.com/win/2004/08/events
xmlns                              : myNs
Id                                 : 404
AssemblyPath                       : C:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Office.Datacenter.ActiveMonitoringLocal.dll
TypeName                           : Microsoft.Office.Datacenter.ActiveMonitoring.OverallXFailuresMonitor
Name                               : Mapi.Submit.Monitor
WorkItemVersion                    : [null]
ServiceName                        : MailboxTransport
DeploymentId                       : 0
ExecutionLocation                  : [null]
CreatedTime                        : 2018-10-03T09:48:32.9036616Z
CreatedTime                        : 2018-10-03T09:48:32.9036616Z
Enabled                            : 1
TargetPartition                    : [null]
TargetGroup                        : [null]
TargetResource                     : MailboxTransport
TargetExtension                    : [null]
TargetVersion                      : [null]
RecurrenceIntervalSeconds          : 0
TimeoutSeconds                     : 30
StartTime                          : 2018-10-03T09:48:32.9036616Z
UpdateTime                         : 2018-10-03T09:45:12.3073447Z
MaxRetryAttempts                   : 0
ExtensionAttributes                : [null]
SampleMask                         : Mapi.Submit.Probe
MonitoringIntervalSeconds          : 3600
MinimumErrorCount                  : 0
MonitoringThreshold                : 8
SecondaryMonitoringThreshold       : 0
MonitoringSamplesThreshold         : 100
ServicePriority                    : 2
ServiceSeverity                    : 0
IsHaImpacting                      : 0
CreatedById                        : 57
InsufficientSamplesIntervalSeconds : 28800
StateAttribute1Mask                : [null]
FailureCategoryMask                : 0
ComponentName                      : ServiceComponents/MailboxTransport/High
StateTransitionsXml                : <StateTransitions>                                       <Transition ToState=”Unrecoverable” TimeoutInSeconds=”0″ />                 </StateTransitions>
AllowCorrelationToMonitor          : 0
ScenarioDescription                : [null]
SourceScope                        : [null]
TargetScopes                       : [null]
HaScope                            : Server
Version                            : 65536

From the output, you look for the SampleMask. Getting the SampleMask will tell you the probe that is being used in the HealthSet query. From there, you can use that value to get the definition of the probe with the following cmdlets:

OUTPUT:

auto-ns2 : http://schemas.microsoft.com/win/2004/08/events
xmlns : myNs
Id : 99
AssemblyPath : C:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Forefront.Monitoring.ActiveMonitoring.Local.Components.dll
TypeName : Microsoft.Forefront.Monitoring.ActiveMonitoring.Transport.Probes.MapiSubmitLAMProbe
Name : Mapi.Submit.Probe
WorkItemVersion : [null]
ServiceName : MailboxTransport
DeploymentId : 0
ExecutionLocation : [null]
CreatedTime : 2019-01-05T03:22:02.4029588Z
Enabled : 1
TargetPartition : [null]
TargetGroup : [null]
TargetResource : [null]
TargetExtension : [null]
TargetVersion : [null]
RecurrenceIntervalSeconds : 300
TimeoutSeconds : 30
StartTime : 2019-01-05T03:23:36.4029588Z
UpdateTime : 2019-01-05T03:17:17.2695414Z
MaxRetryAttempts : 2
ExtensionAttributes :
CreatedById : 57
Account :
AccountDisplayName :
Endpoint :
SecondaryAccount :
SecondaryAccountDisplayName :
SecondaryEndpoint :
ExtensionEndpoints : [null]
Version : 65536
ExecutionType : 0

From there you can view and verify the associated error messages that the probe generated when it was run. According to the previous data output, the probe runs every 300 seconds. You will want to filter your logs based on criteria that you input into the cmdlet when searching the log for the events. Properties include:

  • ServiceName – Identifies the HealthSet used.
  • ResultName – Identifies the probe name. When there are multiple probes for a monitor the name will include the sample mask and the resource that you are verifying.
  • Error – Lists the error returned during the failure.
  • ResultType – Lists the value for the result type: 1 = timeout, 2 = poisoned, 3 = success, 4 = failed, 5 = quarantined, 6 = rejected.

So, based on that information, run the following cmdlet to get the last errors in the event log based on the ResultName (Mapi.Submit.Probe) and ResultType (failure). Since there could be a lot of returned data, I tell the cmdlet to select the first 2 results in the output:

SAMPLE OUTPUT:

ExecutionStartTime : 2018-10-12T04:42:26.4725482Z
ExecutionEndTime   : 2018-10-12T04:42:26.5037975Z
ResultId           : 350715748
ResultName         : Mapi.Submit.Probe
ResultType         : 4
Error              : MapiSubmitLAMProbe finished with CheckPreviousMail failure.
ExecutionContext   : MapiSubmitLAMProbe started. This performs – 1. Submits a new message to Store 2. Checks results from previous Send Mail operation. Sequence # = 636741569280603580. First Run? = False. Previous mail submission to store was successful. Results –  # of previous results: 0.  Could Not Find stages that ran.  Previous SendMail failure –  Mail submitted to Store during the previous run never reached SendAsCheck. This may indicate a latency from Store to Submission Service. Investigating.  Found lower SA latency. Indicates an issue in Submission service. Investigate. In SendMail –  NotificationID=00000063-0000-0000-0000-00006ab1f5bc Sending mail. SendMail finished. MapiSubmitLAMProbe finished with CheckPreviousMail failure.
FailureContext     : MapiSubmitLAMProbe finished with CheckPreviousMail failure.

Once we have the error, we can begin to investigate what the Responder did to automatically remediate the issue using the following cmdlet:

Now, in this example, I did NOT get any output due to the fact that I am running the query on a server that did NOT have any localized events that had a recent day. The last time this event occurred based on my notes was September 18th, 2018. But based on the screenshot from my research, you should get some similar output to the output in the below picture:

Responder Output

The responder we were looking for is Mapi.Submit.EscalateResponder as suggested by the screenshot above. This type of responder (Escalate) doesn’t make Managed Availability undertake any automatic repairs but is responsible for log notifications in event logs. After getting the correct responder, you would continue to troubleshoot and attempt to remediate the issue(s) that are behind the HealthSet failure.
In my example case, I found that the Health Mailbox used for the probe test was corrupted and had to be rebuilt. Once that mailbox was functional, the probe test ran successfully.

I hope that this will help you in troubleshooting any alerts in your Exchange environment that are HealthSet based. I know for sure that gathering this information has helped me get a grasp on how the Monitoring works and how it can be used to remediate issues.

A big “Thank You” to the following sites that helped provide most of the information that you see posted here:
Exchange 2013 Managed Availability HealthSet Troubleshooting
Managed availability in Exchange 2013/2016

PowerShell Script to log NETLOGON Events 5719 and 5783, then test the Secure Channel to verify connectivity

In my support role, we would get nightly alerts showing disconnection to the PDC from other DCs and Exchange Servers, giving the following events:

DC02
10/31/2018 23:20:28 5719
NETLOGON
This computer was not able to set up a secure session with a domain controller in domain LDLNET due to the following:
The remote procedure call was cancelled.
This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.

ADDITIONAL INFO
If this computer is a domain controller for the specified domain, it sets up the secure session to the primary domain controller emulator in the specified domain. Otherwise, this computer sets up the
secure session to any domain controller in the specified domain.

DC03
10/31/2018 23:18:58 5783
NETLOGON
The session setup to the Windows NT or Windows 2000 Domain Controller \\DC01.LDLNET.ORG for the domain LDLNET is not responsive. The current RPC call from Netlogon on \\DC03 to \\DC01.ldlnet.org has been cancelled.

In order to validate the secure channel, you normally run the nltest command (you can also run the Test-ComputerSecureChannel PowerShell cmdlet) to verify the connectivity to the PDC on the secure channel. The scenario is though that multiple DCs or Exchange servers are having multiple events at a similar time due to a network hiccup that brought the secure channel offline between the two Servers.

Our team at the time was getting a lot of alerts generated and it was taking an inordinate amount of time to validate and test. In an effort to provide an efficient solution for this issue, I compiled a PowerShell ps1 script to first validate the events posted in the past three hours, and then secondly, test all the DCs and Exchange Servers for the Secure Channel Connectivity:

NOTE: This script needs to be run on a server that has the Exchange and Active Directory RSAT tools for PowerShell.

I can’t really put out the output since it will have customer PII, but you will see where it will list the DC/Exchange Server Name, show the events, then run the test. You can then troubleshoot from there. Also, know that the secure channel test will FAIL when run on the PDC Emulator DC. The PDC Emulator cannot run a secure channel test on itself.

Please, if you have any questions or comments, please leave some feedback! Happy Troubleshooting!

Get-Counter cmdlets…

Sometimes you need to check performance counters within Windows for different services or applications. The problem is being able to record the output if needed.
I have been able to take care of this through PowerShell so that you can get an average of any performance counter output you need over a time period.

According to:  https://blogs.technet.com/b/nexthop/archive/2011/06/02/gpsperfcounters.aspx
A “CookedValue” definition: Performance counters typically have raw values, second values, and cooked values. The raw values and second values are the raw ingredients used by the performance counter, and the “cooked value” is the result of “cooking” those ingredients into something for human consumption. So apparently the CookedValue is the result of combining the counter’s raw data to get a usable value that you can understand and work with.

Here are some examples for Windows:

Examples for Exchange Server:

A couple of links to listings of Performance Counters For Exchange:

https://www.poweradmin.com/help/pa-file-sight-7-1/howto_monitor_exchange.aspx

https://technet.microsoft.com/en-us/library/ff367923(v=exchg.141).aspx

Now, there are more counters available for all types of Windows Applications. You should be able to use every counter that is listed in Performance Monitor on the server you are running the test from.

You can always use the following command to get a list of counters on your server and save them to a file called perfcounters.txt in the C:\Files directory:

I will not go into too much detail as of now, but I will probably update this as I get more information and comments on the post.
Again, this blog is for quick reference and usage when doing reactive support. As this blog grows, I will add more in depth information. Don’t hesitate though to contact me with your questions and comments.

Getting all Exchange Databases listed and whether or not they are on their preferred node or not.

This is a great one liner in PowerShell that will allow you to get a listing of all the databases for your Exchange Server environment. It will also tell you if those databases are on their preferred node in the DAG and whether they are actively mounted on that node.

This is helpful to know if you have multiple database fail-overs and need to know which databases are where so that you can re-balance them properly. If you are in a large environment, this will help you get a handle on the issue and be able to remediate quickly.

Here is an example of the output:

Now, that you have your listing of DBs and their status, you can run the following script from PowerShell to mount those DBs to their preferred nodes:

Since SLA and remediation are big factors in reactive support, having these scripts help save the day when things get quirky in Exchange. Please comment and submit your scripts as well!

Getting Drive Space Through PowerShell for a Server

This cmdlet will list all your mounted volumes, their size, the file system used, and the available free space. You can modify the code to have a where-object statement: ? {$_.Name -like “*logs*”}. This helps if you have an exchange server that has multiple database volumes for DBs and logs and need to quickly find which volume is the culprit.

I also use a lot of these scripts to gather the information quickly so that I can post the output into my incidents that I am working. It’s good to have these handy.

Here is an example output:

NameFree, GBFree, %Capacity, GBFS
C:\ExchangeDB\DAG2DB01\DB\456.8037.451,219.873NTFS
C:\ExchangeDB\DAG2DB01\LOG\39.4999.0339.873NTFS