Generate Automated E-Mail From Performance Counter Alert

Recently, I was trying to load this IT Blog and noticed that the page took a very long time to load properly. So, first thing I did was logon to my Web Server to check things and saw that the Performance was very slow, almost to a halt. When I was able to pull up Server Manager, I saw the following:

The Performance Alert was pegged at 99+ alerts. I went into the alerts for the performance and saw the following:

My CPU Usage was pegged at 100% for many hours. As I was having issues with the server and could not dig any deeper to find what was running to cause the error other the the w3wp process, I decided to reboot the server.
Rebooting did resolve the issue at hand, but I needed a better way to alert myself when the processor pegged at 95% or more over at least a steady two minute period. That way I could keep the websites going and restart the server if necessary due to an issue with a process pegging the CPU.

I don’t have SCCM or any robust alerting software like we have a my employer, but not to worry, Windows Server has a built in method to generate a task using Task Scheduler when the Performance Counter is above threshold for the time period that you set.

Step 1 – Create your PowerShell Script

The first thing that you need to do is write a Powershell script that when run can send an email. While researching this I discovered many ways to accomplish this task, feel free though to experiment and use what is right for your environment.

Here is an example of a Powershell script that when used in conjunction with Task Scheduler and Perfmon can send an email alert automatically when any user defined performance counter threshold condition is met. In my environment I set this to C:\PowerShell\AlertEMail.ps1

If you notice, this Powershell script takes four arguments and assigns them to variables used in the output. It also saves the computer name to a variable which is also used as part of the output. By doing this the script can be used to send an email on any Perfmon Alerting Counter and on any server without additional customization of the script.

Step 2 – Setup a Scheduled Task To Be Run

In Task Scheduler we are going to Create a new Task as show in the following screen shots.

Give the Task a name, you will need to remember it for the next step. Make sure the task is run with an administrative account and is run whether the user is logged on or not.

There are no Triggers to setup here. This Task will actually be triggered through the Perfmon Counter Alert we will set up in Step 3.

On the Action tab you want to define a new action. The action will be to Start a Program and use the following inputs, please adjust for your specific environment.

Program/Script: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Add Arguments: -File C:\PowerShell\AlertEMail.ps1 $(Arg0)

Next, set your Conditions and Settings Tabs to the following settings as I did for this purpose. If need be, you might need to set them differently, but these settings should work just fine.

Step 3 – Create the Alert Trigger in Performance Monitor

Open Performance Monitor and Create a new Data Collector Set under Data Collector Sets > User Defined

I needed to monitor the Total % Processor Time for all processors on the server combined, so I set the counter to monitor from the available counters.

I then set the threshold to Alert when the processor is over 95% with Processor Time. I also let it run as the System account, and then saved the settings:

Once you have created the Data Collector Set go into the Properties of it and make sure the Alerting threshold and Sample Interval is set properly for each Performance Counter. Keep in mind, if you sample every 10 seconds then you should expect to receive an email every 10 seconds as long as the performance counter exceeds the threshold you set. I set my Sample Interval to 2 minutes so that if the percentage stays above 95% over that long of a time interval, I should take a look at the server and see what process is pegging the CPU:

If you select the Log an entry in the application event log don’t expect to see any entries in the normal Application event log. It will be written to the Microsoft-Windows-Diagnosis-PLA/Operational log in the Application and Services log directory.

Finally, we have to set an Alert Task that will trigger the Scheduled Task (AlertEMail) that we created in Step 2. You see that we also pass some of the Task arguments which are used by the PowerShell script to customize the email with the exact error condition associated with the Alert:

Once the Data Collector is configured properly you will want to start it.

If you configured everything correctly you should start receiving emails any time that alert threshold is surpassed. I also manually ran the script once from PowerShell with no Argument Data just to make sure the script would generate the email to my E-Mail Address.

Step 4 – Set the Data Collector Set to run upon server startup.

You now have one more step. Whenever you reboot a server the Perfmon Counter Alert will not start automatically. In order to survive a reboot you must run the following at a command prompt.

Note: “Processor Usage Alert” referenced in the script below is the name of my user defined Data Collector Set.

I now have a simple way to monitor my Web Server should this event happen again with the processor being pegged over a two minute period.

PLEASE COMMENT! I LOVE IMPROVEMENTS TO MY SCRIPTS! HAPPY TROUBLESHOOTING!

Reference:
Setup E-Mail Generation from a Performance Counter Alert