Update Edge Server Certificate in a Hybrid Exchange Environment

LDLNET LLC Banner
LDLNET LLC – Life In Action! Your Source for Professional IT Services!

At work, our group was updating the Exchange Edge Server certificates and having mail flow problems causing messages to be in the Poison Queue and not transfer to Office365 properly. We finally got the procedure down to where it started working. I wanted to post that procedure here since I had never really worked with Edge Servers in the past. If this post can help you in the future, then “I done good!”

Now, everywhere I had read said that you have to remove and then re-create the Edge Subscription between your Transport Servers and the Edge Servers when changing the certificate.

Here is why:
When we subscribe the edge server, an AD LDS account called the EdgeSync Bootstrap Replication Account (ESBRA) is created. This is created using the default certificate private key of the certificate assigned to SMTP service as default, hence as long as we have that certificate the transport servers will be able to authenticate to the Edge server and replicate the required information to ADAM database.

Now when we install a third party certificate we assign SMTP service to it and overwrite the current certificate, basically we change the default SMTP certificate. So, by doing this, the current Edge Subscription will fail as the Edge server will not be able to decrypt the ESRA account passed on when communicating with the transport servers using the new certificate private key.

So, once you have your new 3rd party certificate, you install it to your edge servers:

Then, you enable the Exchange Certificate to be used for SMTP:

Mail flow will be broken at this point. Since messages were going to the poison queue due to the ESBRA account encryption failing when authenticating with the internal Transport Servers, I had to completely stop transport by disabling the Send Connectors between the internal Transport Servers and the Edge servers from the Transport Server.

The configuration of the Edge Servers were that there were two servers in the Edge Farm. Since one of the servers had not had a proper sync in a while, I decided to remove the recipient database that had been replicated to the failing server when removing the Edge Subscription. The other server, I left the recipient database in place so that we could get one server up and running quickly since transport was stopped at this point.

Here is the command that was run to remove the Edge Subscriptions. This needed to be completed on both the Edge Servers and the corresponding Transport Server:

I then had to create a new Edge Subscription file on each Edge Server to copy to the Transport Server. I already had connectors set so I did not need to recreate those connectors.

I copied the xml files of each Edge Server to the Transport Server and ran the following cmdlet to create the Edge Subscription to the Edge Servers. I then had the Edge Servers Rebooted for good measure before redoing a Full Manual Edge Sync.

I next had to preform a full manual EdgeSync from the transport server to the Edge Servers to assure that the recipient database on the AD LDS instance was up to date and that the send connectors were replicated properly.

I next had to re-run the Hybrid Configuration Wizard so that I could configure the Edge Servers as the transport for Hybrid cloud-bound Messages. Once the Edge Servers were chosen to transport Hybrid cloud-bound messages, I selected the new Edge Certificate so that transport would work properly when re-enabled and O365 would recognize the new certificate for Hybrid messages bound for the cloud.

I next re-enabled the Edge Send Connectors so that mail flow would begin working once the Full Edge Synchronization was completed. You have to let that complete before you can begin mail flow again so that messages won’t be delivered to the Poison Queue.

Mail flow began working. It took about 90 minutes for all the queues to clear properly that had queued messages waiting to transport. Any Poison Queued messages were removed with NDRs sent to the senders.

It was a doozy to say the least. Happy Troubleshooting!
Leave Comments or Questions you may have!

References:
Exchange 2010 Edge Transport Server: Configuring EdgeSync
Mail flow breaks after renewing SSL Certificate on Edge server with Edge Subscription
Start-EdgeSynchronization

PowerDNS Script

I was compiling some scripts to be able to modify DNS records in my previous post. While browsing through different scripts in the TechNet Gallery, I came across the following Script that provides a menu, options, and different settings which really make it a great script to use if you do a lot of DNS Modification and want to do it through PowerShell.

Here is the link to the original script page, but I have updated and modified the script to include being able to add/remove DNS Zones as well.

The DNS Zone functions have not been tested as of yet. I still have to get on my server farm at home and run this. It will save time though with me having to switch servers when adding a bulk list of DNS zones for my website farm. Play with the script and let me know what you think!

Removing a DNS Record through Powershell

In most environments, an admin usually just jumps on the server that they need to work from and does their work from there. An example of this would be an admin working on an IIS Web server and needing to remove a DNS A record from DNS without having to logon to the DNS server itself so that they can quickly make their changes in IIS.

A quick way to do this would be to run the following ps1 script in PowerShell in order to be able to remove the record quickly:

Sample Output from the script.
Sample Output from the Script removing DNS A Record: test.ldlnet.local

Now this works for a single DNS A Record. If there are multiple IPs for the same DNS record, for example, test.ldlnet.local points to both 192.168.1.23 and 192.168.1.24, then you probably need to run the following script listed here to keep the script from failing with an error. I have also expanded the entries to help the input be more specific:

Output from RemoveDNSRecord.ps1 for removing DNS A Record test.ldlnet.local with IP of 192.168.1.24

I have found some other good scripts that I will post to the blog to help manage DNS records through PowerShell. This should get things started for now. Happy Troubleshooting!

How to log off a RDP session remotely.

Have you ever tried to logon to a Remote Desktop session on a Windows Server and you get stuck on the following screen?

Stuck Logging Off

Well, here is a simple way you can remotely kill that RDP session through PowerShell so that you can logon to the server again…

Sample Output:

Output from qwinsta command…

Once you get the session ID, you can run the following to kick off the user’s session completely so that you can log into the server again:

Note: The session will be completely removed from RDP and anything running will be lost, but most of the time, you don’t have to worry about losing anything as the whole reason to lose the session is because you cannot logoff of it normally.

Life is then good again as you can log into your RDP session. Yay!

MaxConcurrentAPI Script for Netlogon Issues

I get incidents from time to time that deal with Netlogon Service Issues. For example: Semaphore Waiters, Semaphore Timeouts, Semaphore Acquires, etc…

Here is a script I got from the Microsoft Gallery
In some enterprise environments the sheer volume of NTLM authentication can produce performance bottlenecks on servers. To help make the problem easier to detect, this PowerShell script was written.

Execution:

Now, I modified this script taking out the clear screen parameter so that I could be run against multiple servers. Place the script in your Scripts directory and name it CheckMaxConcurrentApiScript.ps1

First, in PowerShell, gather your list of servers:

Or

Next, run the command to run the ps1 against those servers:

Or

Sample Output:

DC03
Detection Time : 12/13/2018 7:56:16 PM
Problem Detected : False
Server Name : DC03
Server Role : Domain Controller
Domain Name : ldlnet.org
Operating System : Microsoft Windows Server 2008 R2 Enterprise
Time Since Last Reboot : 4 days 22 hours
Current Effective MaxConcurrentApi Setting : 10
Suggested MaxConcurrentApi Setting (may be same as current) : 10
Current Threads in Use (Semaphore Holders) : 0
Clients Currently Waiting (Semaphore Waiters) : 0
Cumulative Client Timeouts (Semaphore Timeouts) : 17
Cumulative MaxConcurrentApi Thread Uses (Semaphore Acquires) : 3493999
Duration of Calls (Avg Semaphore Hold Time) : 0

EXCH02
Detection Time : 12/13/2018 8:00:53 PM
Problem Detected : False
Server Name : EXCH02
Server Role : Member Server
Domain Name : ldlnet.org
Operating System : Microsoft Windows Server 2008 R2 Standard
Time Since Last Reboot : 4 days 23 hours
Current Effective MaxConcurrentApi Setting : 10
Suggested MaxConcurrentApi Setting (may be same as current) : 10
Current Threads in Use (Semaphore Holders) : 0
Clients Currently Waiting (Semaphore Waiters) : 0
Cumulative Client Timeouts (Semaphore Timeouts) : 570
Cumulative MaxConcurrentApi Thread Uses (Semaphore Acquires) : 1682257
Duration of Calls (Avg Semaphore Hold Time) : 0

Hopefully, this script will assist you with gathering the needed information to help you balance the netlogon load between your servers when needed in your environment.

HAPPY TROUBLESHOOTING!

Exchange Back Pressure and Transport Issues

Sometimes you’ll get a situation where your email will stop flowing on one of your Exchange servers. Most of the time, we’re worried about our database and log file drives becoming full, but we don’t necessarily look at the configuration of our Transport servers to see if the resources in those directories become full or taxed to the point where it causes, “Back Pressure”. Exchange has events setup to monitor when that threshold is crossed and transport functionality is hindered:

  • Event ID 15004: Increase in the utilization level for any resource (eg from Normal to Medium)
  • Event ID 15005: Decrease in the utilization level for any resource (eg from High to Medium)
  • Event ID 15006: High utilization for disk space (ie critically low free disk space)
  • Event ID 15007: High utilization for memory (ie critically low available memory)

If you think that your server is experiencing a back pressure event, you can look quickly through event viewer for these events with the following script:

In most cases, you get the 15006 event:

Event 15006, MSExchangeTransport
Microsoft Exchange Transport is rejecting message submissions because the available disk space has dropped below the configured threshold. The following resources are under pressure:
Used disk space (“C:\Microsoft\Exchange Server\V15\TransportRoles\data\Queue”)
Used disk space (“C:\Microsoft\Exchange Server\V15\TransportRoles\data”)

Overall Resources
The following components are disabled due to back pressure:
Mail resubmission from the Message Resubmission component.
Mail submission from Pickup directory
Mail submission from Replay directory
Mail resubmission from the Shadow Redundancy Component
Inbound mail submission from the Internet

Exchange uses the following formula to calculate the threshold at which these events fire:

100 * (hard disk size – fixed constant) / hard drive size

So, in order to get transport running again, get your C: Drive cleared so that back pressure is lifted off of the server and transport can run again. You should then get a 15005 Event:

Log Name: Application 
Source: MSExchangeTransport 
Date: 10/19/2017 2:21:52 PM 
Event ID: 15005 
Task Category: ResourceManager 
Level: Information 
Keywords: Classic 
User: N/A 
Computer: EX01.ldlnet.org 
Description: 
The resource pressure decreased from Medium to Low.No components disabled due to back pressure. 
The following resources are in normal state: 
Private bytes 
System memory 
Version buckets[C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que] 
Jet Sessions[C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que] 
Checkpoint Depth[C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que] 
Queue database and disk space (“C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que”) 
Used disk space (“C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue”) 
Used disk space (“C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data”) 
Overall Resources

Now that you’ve completed this, how to do setup your Exchange Environment to keep this from happening again? Well, I would pick a drive volume that you’d never have to worry about filling up, or give the transport its own drive volume. This can be accomplished with a .ps1 script that is installed in the default Scripts directory on your Exchange Server installation:

‘C:\Program Files\Microsoft\Exchange Server\Vxx\scripts’

The name of that file is Move-TransportDatabase.ps1 and it
changes the location of the transport directories, moves the Queue Database and restarts the Transport service automatically. Here is an example of how the script is executed when running Exchange PowerShell with elevated privileges and wanting to move all the services to the E: Drive:

So, that’s how you get your transport directories configured to relieve “back pressure”. In my experience, somebody was doing a PST export of a mailbox to the local C: Drive instead of a specific drive volume that wouldn’t affect the OS, Exchange, and Transport. That’s for another time though! Happy Troubleshooting!

Reference: Exchange 2016 – Back Pressure
Reference: A Guide To Back Pressure.
Reference: Change Exchange Server 2013/2016 Mail Queue Database Location

Running Test-MailFlow on remote Exchange Servers

In my job I try to make the process as efficient as possible so that I can determine the issue quickly and then resolve it as quickly as possible. I was having issue with the Test-Mailflow cmdlet and running it remotely against the servers. I was getting the following error:

MapiExceptionSendAsDenied: Unable to submit message. (hr=0x80070005, ec=1244)

If I had multiple servers to test, I would have to logon to each server and run the test which is not efficient at all. I wanted to automate it more without having to change permissions to do so. I wanted to run an Invoke-Command and place the PSSession for Exchange in that command so that I could run the Test-Mailflow cmdlet and get the results.

Paul Cunningham wrote a great article and script to resolve this. READ HERE

His script allows you to input the server name when running the PS1 from the PowerShell Command Prompt:

I was able to take the Test-MailflowRemote.ps1 script and set it to run on all the mailbox servers for the environment I was monitoring. Now, we can only run the Test-MailFlow cmdlet against Exchange Mailbox Servers that have active databases mounted on them. So, I run the following first to get the list of Mailbox Servers that contain at least 1 active database:

I then run the ps1 script using the array I created with the $Svrs variable:

Output:

This helps a bunch when you need to run on multiple servers and get the test information quickly. Please comment! Happy Troubleshooting!

Protected AD Groups and the problems they can cause accounts

I have run into this issue over the years with accounts being in the Domain Admins group and having issues running PowerShell cmdlets as well as not being able to connect to ActiveSync from a mobile device with the account.

These issues are due to the AdminSDHolder Template in AD and the SDProp Process that is run every 60 Minutes in AD.
This is explained in fantastic detail through the following Microsoft article: Protected Accounts & Groups In Active Directory

Here is an example of an issue that occurred in one of the environments that I was managing. A user was trying to run the following AD cmdlet in PowerShell on DC01:

The user got the following error when the cmdlet was executed:

Set-ADUser : Insufficient access rights to perform the operation
At line:1 char:1
+ Set-ADUser lancel -Server dc01.ldlnet.org -Replace @{title=”Senior O …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo: NotSpecified: (lancel:ADUser) [Set-ADUser], ADException
+ FullyQualifiedErrorId : ActiveDirectoryServer:8344,Microsoft.ActiveDirectory.Management.Commands.SetADUser

The issue was that the admin account used to run the cmdlet was in the Domain Admins group and was not inheriting permissions per the AdminSDHolder template that was applied to the account:

I checked to see that the admin account was in a protected group:

I next went to the Security Tab > Advanced Button and saw that the Enable Inheritance button was visible:

I’ve circled where to look in the window.

This verifies that the account is protected due to being in the Domain Admins group. Now, there are two workarounds for this particular error that we were experiencing.

  1. Click the Enable Inheritance button. This will cause the permissions to be inherited temporarily. When SDProp is cycled again, the account will lose any inherited permissions and will be essentially “broken” again. This is not good if you’re going to be running cmdlets regularly to modify AD Accounts.
  2. The preferred method to work around this issue is to set the -Server parameter to point to a different DC than the one you are on. So, essentially, we tell the cmdlet to execute on DC02 when running the cmdlet from DC01.

Either method will allow the cmdlet to execute successfully and modify the object. You would think that Microsoft would have noticed this issue with running an admin cmdlet for Active Directory, but they have not fixed this issue as of yet nor do i think they plan to. I would just go with workaround number two and remain sane.

Another example of this Protected Group issue comes with an account in a Protected Group that has a mailbox not being able to connect to Exchange ActiveSync when setting up their mobile device.

  • You usually get a 500 error on the device that you cannot connect.
  • You will also see event 1053 in Event Viewer alluding to not having sufficient access to create the container for the user in AD.

Read this page for more information: Exchange ActiveSync Permissions Issue with Protected Groups

So, in your endeavors admins, keep this in mind when running into these types of problems. Happy Troubleshooting!

Exchange Server HealthSets

This is a monitoring feature included with Exchange that until recently, I did not know existed, as it wasn’t really mentioned in any of my dealings with Exchange Server up until recently. The HealthSets feature monitors every aspect of a running Exchange Server and is broken down into three monitoring components:

  • Probe: used to determine if Exchange components are active.
  • Monitor: when probes signal a different state then the one stored in the patters of the monitoring engine, monitoring engine will determine whether a component or feature is unhealthy.
  • Responder: will take action if a monitor will alert the responder about an unhealthy state. Responders will take different actions depending on the type of component or feature. Actions can start with just recycling the application pool and can go to as far as restarting the server or even worse putting the server offline so it won’t accept any connections.

From what I have experienced in the past year with these HealthSets, an alert will be thrown due to a change in a service, or a restart of a service, a failed monitoring probe result, or something of the like. The healthset will become “unhealthy” in state at that time. You can run the following on a server in order to get the healthset status of that server:

If you get alerts for multiple Exchange Servers, let’s say for instance, the transport array, you can run the following cmdlets to get the status of all the Transport Servers in the array:

HealthSet PowerShell Output
HealthSet PowerShell Output

Now, a lot of times, the Unhealthy value in the HealthSet will have corrected itself as per the Responder, even though the AlertValue will remain Unhealthy. To clear the cache quickly and have the monitor probes run again for verification, perform the following restarts of services from this cmdlet in this order:

That should clear the probe results and let them run again. Now, should they again return an error, we will need to dig deeper to figure out the issue.
What you will want to do first is get the monitor definition. In this example, the Mapi.Submit.Monitor was the component that was unhealthy in the healthset. I had to run the following cmdlet to get the Monitor Definition:

Output:

auto-ns2                           : http://schemas.microsoft.com/win/2004/08/events
xmlns                              : myNs
Id                                 : 404
AssemblyPath                       : C:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Office.Datacenter.ActiveMonitoringLocal.dll
TypeName                           : Microsoft.Office.Datacenter.ActiveMonitoring.OverallXFailuresMonitor
Name                               : Mapi.Submit.Monitor
WorkItemVersion                    : [null]
ServiceName                        : MailboxTransport
DeploymentId                       : 0
ExecutionLocation                  : [null]
CreatedTime                        : 2018-10-03T09:48:32.9036616Z
CreatedTime                        : 2018-10-03T09:48:32.9036616Z
Enabled                            : 1
TargetPartition                    : [null]
TargetGroup                        : [null]
TargetResource                     : MailboxTransport
TargetExtension                    : [null]
TargetVersion                      : [null]
RecurrenceIntervalSeconds          : 0
TimeoutSeconds                     : 30
StartTime                          : 2018-10-03T09:48:32.9036616Z
UpdateTime                         : 2018-10-03T09:45:12.3073447Z
MaxRetryAttempts                   : 0
ExtensionAttributes                : [null]
SampleMask                         : Mapi.Submit.Probe
MonitoringIntervalSeconds          : 3600
MinimumErrorCount                  : 0
MonitoringThreshold                : 8
SecondaryMonitoringThreshold       : 0
MonitoringSamplesThreshold         : 100
ServicePriority                    : 2
ServiceSeverity                    : 0
IsHaImpacting                      : 0
CreatedById                        : 57
InsufficientSamplesIntervalSeconds : 28800
StateAttribute1Mask                : [null]
FailureCategoryMask                : 0
ComponentName                      : ServiceComponents/MailboxTransport/High
StateTransitionsXml                : <StateTransitions>                                       <Transition ToState=”Unrecoverable” TimeoutInSeconds=”0″ />                 </StateTransitions>
AllowCorrelationToMonitor          : 0
ScenarioDescription                : [null]
SourceScope                        : [null]
TargetScopes                       : [null]
HaScope                            : Server
Version                            : 65536

From the output, you look for the SampleMask. Getting the SampleMask will tell you the probe that is being used in the HealthSet query. From there, you can use that value to get the definition of the probe with the following cmdlets:

OUTPUT:

auto-ns2 : http://schemas.microsoft.com/win/2004/08/events
xmlns : myNs
Id : 99
AssemblyPath : C:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Forefront.Monitoring.ActiveMonitoring.Local.Components.dll
TypeName : Microsoft.Forefront.Monitoring.ActiveMonitoring.Transport.Probes.MapiSubmitLAMProbe
Name : Mapi.Submit.Probe
WorkItemVersion : [null]
ServiceName : MailboxTransport
DeploymentId : 0
ExecutionLocation : [null]
CreatedTime : 2019-01-05T03:22:02.4029588Z
Enabled : 1
TargetPartition : [null]
TargetGroup : [null]
TargetResource : [null]
TargetExtension : [null]
TargetVersion : [null]
RecurrenceIntervalSeconds : 300
TimeoutSeconds : 30
StartTime : 2019-01-05T03:23:36.4029588Z
UpdateTime : 2019-01-05T03:17:17.2695414Z
MaxRetryAttempts : 2
ExtensionAttributes :
CreatedById : 57
Account :
AccountDisplayName :
Endpoint :
SecondaryAccount :
SecondaryAccountDisplayName :
SecondaryEndpoint :
ExtensionEndpoints : [null]
Version : 65536
ExecutionType : 0

From there you can view and verify the associated error messages that the probe generated when it was run. According to the previous data output, the probe runs every 300 seconds. You will want to filter your logs based on criteria that you input into the cmdlet when searching the log for the events. Properties include:

  • ServiceName – Identifies the HealthSet used.
  • ResultName – Identifies the probe name. When there are multiple probes for a monitor the name will include the sample mask and the resource that you are verifying.
  • Error – Lists the error returned during the failure.
  • ResultType – Lists the value for the result type: 1 = timeout, 2 = poisoned, 3 = success, 4 = failed, 5 = quarantined, 6 = rejected.

So, based on that information, run the following cmdlet to get the last errors in the event log based on the ResultName (Mapi.Submit.Probe) and ResultType (failure). Since there could be a lot of returned data, I tell the cmdlet to select the first 2 results in the output:

SAMPLE OUTPUT:

ExecutionStartTime : 2018-10-12T04:42:26.4725482Z
ExecutionEndTime   : 2018-10-12T04:42:26.5037975Z
ResultId           : 350715748
ResultName         : Mapi.Submit.Probe
ResultType         : 4
Error              : MapiSubmitLAMProbe finished with CheckPreviousMail failure.
ExecutionContext   : MapiSubmitLAMProbe started. This performs – 1. Submits a new message to Store 2. Checks results from previous Send Mail operation. Sequence # = 636741569280603580. First Run? = False. Previous mail submission to store was successful. Results –  # of previous results: 0.  Could Not Find stages that ran.  Previous SendMail failure –  Mail submitted to Store during the previous run never reached SendAsCheck. This may indicate a latency from Store to Submission Service. Investigating.  Found lower SA latency. Indicates an issue in Submission service. Investigate. In SendMail –  NotificationID=00000063-0000-0000-0000-00006ab1f5bc Sending mail. SendMail finished. MapiSubmitLAMProbe finished with CheckPreviousMail failure.
FailureContext     : MapiSubmitLAMProbe finished with CheckPreviousMail failure.

Once we have the error, we can begin to investigate what the Responder did to automatically remediate the issue using the following cmdlet:

Now, in this example, I did NOT get any output due to the fact that I am running the query on a server that did NOT have any localized events that had a recent day. The last time this event occurred based on my notes was September 18th, 2018. But based on the screenshot from my research, you should get some similar output to the output in the below picture:

Responder Output

The responder we were looking for is Mapi.Submit.EscalateResponder as suggested by the screenshot above. This type of responder (Escalate) doesn’t make Managed Availability undertake any automatic repairs but is responsible for log notifications in event logs. After getting the correct responder, you would continue to troubleshoot and attempt to remediate the issue(s) that are behind the HealthSet failure.
In my example case, I found that the Health Mailbox used for the probe test was corrupted and had to be rebuilt. Once that mailbox was functional, the probe test ran successfully.

I hope that this will help you in troubleshooting any alerts in your Exchange environment that are HealthSet based. I know for sure that gathering this information has helped me get a grasp on how the Monitoring works and how it can be used to remediate issues.

A big “Thank You” to the following sites that helped provide most of the information that you see posted here:
Exchange 2013 Managed Availability HealthSet Troubleshooting
Managed availability in Exchange 2013/2016

PowerShell Script to log NETLOGON Events 5719 and 5783, then test the Secure Channel to verify connectivity

In my support role, we would get nightly alerts showing disconnection to the PDC from other DCs and Exchange Servers, giving the following events:

DC02
10/31/2018 23:20:28 5719
NETLOGON
This computer was not able to set up a secure session with a domain controller in domain LDLNET due to the following:
The remote procedure call was cancelled.
This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.

ADDITIONAL INFO
If this computer is a domain controller for the specified domain, it sets up the secure session to the primary domain controller emulator in the specified domain. Otherwise, this computer sets up the
secure session to any domain controller in the specified domain.

DC03
10/31/2018 23:18:58 5783
NETLOGON
The session setup to the Windows NT or Windows 2000 Domain Controller \\DC01.LDLNET.ORG for the domain LDLNET is not responsive. The current RPC call from Netlogon on \\DC03 to \\DC01.ldlnet.org has been cancelled.

In order to validate the secure channel, you normally run the nltest command (you can also run the Test-ComputerSecureChannel PowerShell cmdlet) to verify the connectivity to the PDC on the secure channel. The scenario is though that multiple DCs or Exchange servers are having multiple events at a similar time due to a network hiccup that brought the secure channel offline between the two Servers.

Our team at the time was getting a lot of alerts generated and it was taking an inordinate amount of time to validate and test. In an effort to provide an efficient solution for this issue, I compiled a PowerShell ps1 script to first validate the events posted in the past three hours, and then secondly, test all the DCs and Exchange Servers for the Secure Channel Connectivity:

NOTE: This script needs to be run on a server that has the Exchange and Active Directory RSAT tools for PowerShell.

I can’t really put out the output since it will have customer PII, but you will see where it will list the DC/Exchange Server Name, show the events, then run the test. You can then troubleshoot from there. Also, know that the secure channel test will FAIL when run on the PDC Emulator DC. The PDC Emulator cannot run a secure channel test on itself.

Please, if you have any questions or comments, please leave some feedback! Happy Troubleshooting!

VCSServiceManager.msi

LDLNET LLC - Life In Action! Your Source for Professional IT Services!

Let me just say that file is possibly: THE WORST FILE THAT EVER EXISTED ON THE FACE OF THIS EARTH! 

I have had the absolute worst experience trying to install VMWare vCenter Server 6.x onto my Windows Server at the house. I didn’t really have any issues with installing vSphere 6.7 on my HPE Proliant DL360 g6 server so that I could update my environment. I was going to put vCenter on as well so that I could keep my servers updated and such.

The main issue with the install is that it fails when installing the VCSServiceManager.msi component with a 1603 error. Needless to say, this has been a major issue that started with version 6.0. I have looked through all of the following articles in my search to remediate this issue:

I’ve looked through more than that honestly and was NOT able to get past that part of the install, now I did install the msi separately, and the main install acted like it finished successfully, but none of the services installed on the server and the vCenter Component was basically non existent when trying to access it. I even went through setting up an Microsoft SQL Server Express back end thinking it might be an issue with the Python SQL component that comes with vCenter. I even formatted my Server OS and started from scratch. No luck what so ever on the installation. I bet I spent 30 man hours over the course of a week or two trying install after install.

Seeing that it is the evaluation version, I cannot seek direct help from VMWare as their support is paid support, but being an IT guy myself, and seeing all the problems with that part of the install, leads me to believe that I wasn’t the problem when I was installing the software. I followed all the processes to get the installation to go smoothly and correctly, but did not succeed.

I used to be a real advocate for VMWare as a great Hypervisor, but, I will be migrating to Hyper-V now. I’m running all 2019 servers anyway, I should have Hyper-V on my host.

Question: Has anyone had issues with Hyper-V or Server 2019 Datacenter on an HPE Proliant DL360 g6 server? I haven’t seen too much issue from my research. So if the blog is down this weekend, you’ll have knowledge why as I convert my VMs to VHDX disks and get Hyper-V setup.

Happy Reading! Please leave some feedback!

Getting Certificate Information in PowerShell

When you have certificates expiring, you need to be able to gather the information about the certificates so that you can prepare the renewal requests properly and get the certificate renewed. Now, Windows doesn’t have a native application that is readily available to look up certificate data. You have to open the MMC console and then add the proper Certificate Snap-In to gain access to the certificate store.

In dealing with this, I have found that PowerShell is a great method to be able to gather all of this data quickly and in a way where you can copy/paste the information that you need in order to generate your request properly for a new certificate or a renewal.

First off, you have to make sure that the PKI Module is installed on your system that you are running PowerShell on:

Download and install PowerShell PKI module from the PowerShell Gallery using PowerShell

Module Requirements

  • Windows PowerShell 3.0 or higher
  • .NET Framework 4.0 or higher

This module can run on any of the specified operating systems:

  • Windows Server 2008*/2008 R2*/2012*/2012 R2*/2016*/2019*
  • Windows Vista/7/8/8.1/10

* — Server Core installation is not supported.

NOTE: Module installation requires installed RSAT (Remote System Administration Tools)

Once you have it installed, you can then begin accessing the Certificate Store on the server that you are on:

NOTE: Setting the location to LocalMachine\My will place PowerShell in the Personal Store of the Local Computer Account.

The Get-ChildItem cmdlet will return the information of the certificates that are in the directory that you are in. You can also amend the cmdlet with given parameters to get the information from another machine:

To get the properties of all certificates expiring in 120 days locally:

To get the properties of all certificates expiring in 120 days on a remote server:

Now, let’s say that you have certificates expiring in 120 days on all of your CAS Exchange Servers and you need to get the information on all those certificates since they do not have the same thumbprint. You can run the following commands in sequence to be able to get the information from all of those servers:

In another post I will expand on this topic and show how to generate CSRs, Import and Export Certificates, and renew certificates. I’m still doing research on those topics and will compile my information as soon as I can get it organized. Hope this helps!

Check the Crawl Status of a SharePoint Farm

A lot of times in my job, we get alerts for processor usage spikes and memory being below threshold for normal server usage. When these would occur on SharePoint servers, I would go through finding out what processes were running that were using a lot of processor and memory. Turns out that it was usually noderunner.exe that was causing the issue which has to deal with the crawl function of the SharePoint data being indexed by the servers in the farm. So, after looking around on a few sites, I compiled the following ps1 script that can be run on a SharePoint server to get the Crawling status of all the SharePoint Content Sites and how long it took to crawl during the prior cycle:

It will output:
– The name of the Content Site being Crawled
– Whether or not the crawl is idle
– If idle, when the crawl started
– If the last crawl took three hours or less

I hope this will give some help to those deep diving into SharePoint Performance.

Get-Counter cmdlets…

Sometimes you need to check performance counters within Windows for different services or applications. The problem is being able to record the output if needed.
I have been able to take care of this through PowerShell so that you can get an average of any performance counter output you need over a time period.

According to:  https://blogs.technet.com/b/nexthop/archive/2011/06/02/gpsperfcounters.aspx
A “CookedValue” definition: Performance counters typically have raw values, second values, and cooked values. The raw values and second values are the raw ingredients used by the performance counter, and the “cooked value” is the result of “cooking” those ingredients into something for human consumption. So apparently the CookedValue is the result of combining the counter’s raw data to get a usable value that you can understand and work with.

Here are some examples for Windows:

Examples for Exchange Server:

A couple of links to listings of Performance Counters For Exchange:

https://www.poweradmin.com/help/pa-file-sight-7-1/howto_monitor_exchange.aspx

https://technet.microsoft.com/en-us/library/ff367923(v=exchg.141).aspx

Now, there are more counters available for all types of Windows Applications. You should be able to use every counter that is listed in Performance Monitor on the server you are running the test from.

You can always use the following command to get a list of counters on your server and save them to a file called perfcounters.txt in the C:\Files directory:

I will not go into too much detail as of now, but I will probably update this as I get more information and comments on the post.
Again, this blog is for quick reference and usage when doing reactive support. As this blog grows, I will add more in depth information. Don’t hesitate though to contact me with your questions and comments.

Show available RAM on a server

Here is a PowerShell .ps1 file snippet that will output the available RAM on a server.
You can name it freemem.ps1 and place in your local PowerShell scripts directory.
Thanks to the following for the script: Click Here

If you need to find out what processes are using the most memory, you can run the following PowerShell cmdlet to do so:

Happy Troubleshooting!

Getting all Exchange Databases listed and whether or not they are on their preferred node or not.

This is a great one liner in PowerShell that will allow you to get a listing of all the databases for your Exchange Server environment. It will also tell you if those databases are on their preferred node in the DAG and whether they are actively mounted on that node.

This is helpful to know if you have multiple database fail-overs and need to know which databases are where so that you can re-balance them properly. If you are in a large environment, this will help you get a handle on the issue and be able to remediate quickly.

Here is an example of the output:

Now, that you have your listing of DBs and their status, you can run the following script from PowerShell to mount those DBs to their preferred nodes:

Since SLA and remediation are big factors in reactive support, having these scripts help save the day when things get quirky in Exchange. Please comment and submit your scripts as well!

Getting Drive Space Through PowerShell for a Server

This cmdlet will list all your mounted volumes, their size, the file system used, and the available free space. You can modify the code to have a where-object statement: ? {$_.Name -like “*logs*”}. This helps if you have an exchange server that has multiple database volumes for DBs and logs and need to quickly find which volume is the culprit.

I also use a lot of these scripts to gather the information quickly so that I can post the output into my incidents that I am working. It’s good to have these handy.

Here is an example output:

NameFree, GBFree, %Capacity, GBFS
C:\ExchangeDB\DAG2DB01\DB\456.8037.451,219.873NTFS
C:\ExchangeDB\DAG2DB01\LOG\39.4999.0339.873NTFS

Welcome to my IT Blog…

This blog is going to be a dedicated repository linked to pages, posts, documentation, links, and information that I find during my troubleshooting processes within the IT world. Some posts will contain code snippets that you can use with your own work if needed. Feel free to comment and get this thing rolling!

I now have contributors! Please welcome their input as valued colleagues in the IT Industry! Be on the lookout for their blog posts and please comment!

LDLNET LLC – Your source for Professional IT Services