Hyper-V 2019 will NOT mount ISO from a network share.

Like most IT guys. They have a repository of their ISO images saved on a network share so that they can mount the ISO if needed on multiple machines. I recently switched to Hyper-V and have been having an issue with creating VMs and using my ISO from my network share to do so.
Hyper-V Manager available through RSAT doesn’t have an option to mount an ISO or capture a drive from a machine on which is running. Instead it gives you drives of the Hyper-V host, and that would of course require you to have an ISO or the disc itself present on the host. I didn’t want to do that. I would rather have my repository share available for that purpose to allow for all the drive space to be available on the Hyper-V host.

So, I would map a network drive with my ISOs. The mapping would succeed, but mapped drive (letter) will not be visible in Hyper-V manager when trying to mount an ISO. Okay, so next I tried mounting from UNC share directly, but that would also fail, with the message:
“‘VM’ failed to add device ‘Virtual CD/DVD Disk’” “User account does not have permission required to open attachment”.

hyperv1
Access Denied Error when trying to mount the ISO

It goes back to the constrained delegation requirement for the Hyper-V host accounts to be used to perform functions such as this. This has been a pain to say in the least, as I have also had issues with live migration with my machines not being clustered due to different hardware.

So, in researching, I found this blog post. It has helped me through this issue with mapping the shared folder with the ISOs.

The cause of the problem is that the Hyper-V is intended to run with VMM Library Server and to mount files from it, not any random share. To re-mediate this:

  • You need to assign full NTFS and share permissions to computer account of Hyper-V on a shared folder with ISO’s you want to mount.
  • In AD on the computer account of Hyper-V machine delegate specific service ‘cifs’ to the machine you want your ISO’s mounted from. Microsoft calls this constrained delegation.

Here is step by step procedure for the constrained delegation:

  1. Go to Active Directory Users and Computers
  2. Find the Hyper-V server computer account and open up its properties.
  3. Go to Delegation tab.
  4. Select Trust this computer for delegation to the specified services only radio button.
  5. Click the Add button.
  6. Click the Users or Computers… button.
  7. In the Add Services window, click Users or Computers and enter the computer account that will  act as a library server and click OK.
  8. Select the cifs Service Type and click OK.

The resulting setup should look something like this:

Constrained delegation
What the configuration should look like for constrained delegation

I added both the server that contained the ISO images and the server that I run my RSAT tools from just to be safe. I next rebooted the Hyper-V host (that is a requirement).
When the host rebooted, I was able to successfully create the VM.

Hopefully, this will also solve my issue with live migration between my hosts. I will have to test that again and will inform everyone here if that succeeds as well!

PLEASE COMMENT!
THANKS FOR READING!

References:
Hyper-V Server 2012 won’t mount ISO from a network share
Hyper-V authentication in Windows Server 2016 for managing remote Hyper-V servers through RSAT
Constrained Delegation

Exchange Setup Repeatedly Says ‘A Restart from a Previous Installation is Pending’

I have had this issue with EVERY upgrade that I have ever attempted for Exchange Server from 2013 through 2019 CU1. You go to run the setup program and during the prerequisite checks, setup stops. The error listed is:

A restart from a previous installation is pending. Please restart the system and rerun setup.

During the prerequisite checks, Exchange Setup looks in the registry at the following keys:

  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Updates\UpdateExeVolatile
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations

Nine times out of ten, a restart does NOT remediate this error. In order for setup to continue properly, you must do the following:

  • Open regedit: Start > Run > regedit.exe
  • Set the HKLM\SOFTWARE\Microsoft\Updates\UpdateExeVolatile key value to 0 or delete it if present. <– This one is usually NOT present.
  • Delete the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations key.
  • Re-run Setup.

You should now be able to run setup and upgrade your Exchange Server.

PLEASE COMMENT!
HAPPY TROUBLESHOOTING!

References:
A Restart From Another Installation Is Pending
Exchange Setup Fails – A Restart From Another Installation Is Pending
Microsoft Document – A Restart From Another Installation Is Pending

Removing Hidden Devices in Device Manager

As you may have knowledge of, if you are reading my blog. I am currently migrating off of VMWare to Hyper-V. Now, as I convert my machines to Hyper-V, it uses a totally different driver for the Network Card. I am having to rebuild the NIC settings within windows to setup the NIC for the Hyper-V VM to get the machines on the network properly again. The VMWare NIC disables and hides the NIC from the VMWare driver in Device Manager.

What this does is make Windows think it has two active network cards, even though one is disabled and removed/hidden in device manager. So, to clean things within Windows, I have to perform the following procedure to remove the hidden device:

Open PowerShell as Administrator
Next, type the following cmdlet and press Enter:

Next, open Device Manager from the PowerShell Session:

When the Device Manager GUI opens, click the View menu
Click 
Show Hidden Devices
Go to the Device that is hidden, in my case the Network Adapter
Right-Click the Device and select Uninstall

Close the Device Manager GUI and PowerShell session

This cleaned the old hardware drivers off the system and allowed the current Hyper-V NIC to be the only one installed.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!

How to transfer FSMO Roles using PowerShell

A rare weekend post for me! HA! I am currently migrating my server environment from VMWare 6.7 to Server 2019 Hyper-V. I have a separate standalone box that I use for my VM backups and as a tertiary DC. Since I had to shut down my VMs in order to convert them, I needed to quickly move my FSMO roles from the DC Virtual Machine to the Standalone box so things would stay running.

I found this great article on how to do that quickly through PowerShell since it is a pain to go into ADUC, ADDT, and setup an MMC for the Schema snap-in.

When you create a domain, all FSMO roles assigned to the first domain controller in the forest by default. You can transfer FSMO roles from one DC to another both the Active Directory graphics snap-ins and the PowerShell command line. Moving FSMO roles using AD PowerShell has the following benefits:

  • You do not need to connect with a MMC snap-ins to the future role owner;
  • Transferring or seizing FSMO roles does not require a connection to the current or future role owner. You can run AD-PowerShell module cmdlets on a Windows Client or Server running RSAT Tools;
  • To seize the FSMO role (if the current owner is not available), it suffices to use an additional parameter -force.

Import the Active Directory Module Into PowerShell:

To get the current forest level FSMO role owners (Domain Naming Master and Schema Master roles) you can use the following PowerShell cmdlet:

To view domain-wide FSMO roles (Infrastructure Master, PDC Emulator and Relative Identifier Master roles):

Transfer FSMO Roles using PowerShell

To transfer FSMO roles between Active Directory domain controllers, we use the PowerShell cmdlet:
Move-ADDirectoryServerOperationMasterRole

To use the Move-ADDirectoryServerOperationMasterRole cmdlet, you must meet the following requirements:

  • There must be at least one DC with a version of Windows Server 2008 R2 or higher
  • PowerShell version 3.0 or newer
  • Active Directory module (2.0  or newer)

NOTE: Unlike the Ntdsutil.exe utility, the Move-ADDirectoryServerOperationMasteRole cmdlet can be performed from any domain computer to migrate the Operations Master roles if you have the appropriate rights (Domain admins and Enterprise Admins).

Import the AD Module:

I needed to move all the roles from one server to the other, so, I ran the following to do so:

NOTE: To simplify the command, you can replace the names of roles with numbers from 0 to 4. The correspondence of names and numbers is given in the table:

PDCEmulator0
RIDMaster1
InfrastructureMaster2
SchemaMaster3
DomainNamingMaster4

So, by having knowledge of these numbers, you can simplify your cmdlet:

NOTE: In the event that the current owner of one or all of the FSMO roles fails, the forced transfer of FSMO roles is performed by the same command, but with the -Force option. Also, after the FSMO roles have been seized, the domain controller from which the roles was seized should never be connected to the domain. You will need to preform a metadata cleanup of the Schema before even thinking about putting that failed server back into production.

Once completed, I ran the previous cmdlets of Get-ADForest and Get-ADDomain to verify that the FSMO roles moved to the destination server.

As of now, my conversion to Hyper-V is going smoothly, although it takes quite a bit of time to convert the hard disks. Thanks again!

HAPPY TROUBLESHOOTING! KEEP SCRIPTING!
PLEASE COMMENT!

Reference:
How To Transfer FSMO Roles Using PowerShell

Generate Automated E-Mail From Performance Counter Alert

Recently, I was trying to load this IT Blog and noticed that the page took a very long time to load properly. So, first thing I did was logon to my Web Server to check things and saw that the Performance was very slow, almost to a halt. When I was able to pull up Server Manager, I saw the following:

The Performance Alert was pegged at 99+ alerts. I went into the alerts for the performance and saw the following:

My CPU Usage was pegged at 100% for many hours. As I was having issues with the server and could not dig any deeper to find what was running to cause the error other the the w3wp process, I decided to reboot the server.
Rebooting did resolve the issue at hand, but I needed a better way to alert myself when the processor pegged at 95% or more over at least a steady two minute period. That way I could keep the websites going and restart the server if necessary due to an issue with a process pegging the CPU.

I don’t have SCCM or any robust alerting software like we have a my employer, but not to worry, Windows Server has a built in method to generate a task using Task Scheduler when the Performance Counter is above threshold for the time period that you set.

Step 1 – Create your PowerShell Script

The first thing that you need to do is write a Powershell script that when run can send an email. While researching this I discovered many ways to accomplish this task, feel free though to experiment and use what is right for your environment.

Here is an example of a Powershell script that when used in conjunction with Task Scheduler and Perfmon can send an email alert automatically when any user defined performance counter threshold condition is met. In my environment I set this to C:\PowerShell\AlertEMail.ps1

If you notice, this Powershell script takes four arguments and assigns them to variables used in the output. It also saves the computer name to a variable which is also used as part of the output. By doing this the script can be used to send an email on any Perfmon Alerting Counter and on any server without additional customization of the script.

Step 2 – Setup a Scheduled Task To Be Run

In Task Scheduler we are going to Create a new Task as show in the following screen shots.

Create Task

Give the Task a name, you will need to remember it for the next step. Make sure the task is run with an administrative account and is run whether the user is logged on or not.

There are no Triggers to setup here. This Task will actually be triggered through the Perfmon Counter Alert we will set up in Step 3.

On the Action tab you want to define a new action. The action will be to Start a Program and use the following inputs, please adjust for your specific environment.

Program/Script: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
Add Arguments: -File C:\PowerShell\AlertEMail.ps1 $(Arg0)

Next, set your Conditions and Settings Tabs to the following settings as I did for this purpose. If need be, you might need to set them differently, but these settings should work just fine.

Step 3 – Create the Alert Trigger in Performance Monitor

Open Performance Monitor and Create a new Data Collector Set under Data Collector Sets > User Defined

Name the Alert (This one is for the Processor Usage)
Assign it as a Performance Counter Alert

I needed to monitor the Total % Processor Time for all processors on the server combined, so I set the counter to monitor from the available counters.

I then set the threshold to Alert when the processor is over 95% with Processor Time. I also let it run as the System account, and then saved the settings:

Once you have created the Data Collector Set go into the Properties of it and make sure the Alerting threshold and Sample Interval is set properly for each Performance Counter. Keep in mind, if you sample every 10 seconds then you should expect to receive an email every 10 seconds as long as the performance counter exceeds the threshold you set. I set my Sample Interval to 2 minutes so that if the percentage stays above 95% over that long of a time interval, I should take a look at the server and see what process is pegging the CPU:

If you select the Log an entry in the application event log don’t expect to see any entries in the normal Application event log. It will be written to the Microsoft-Windows-Diagnosis-PLA/Operational log in the Application and Services log directory.

Finally, we have to set an Alert Task that will trigger the Scheduled Task (AlertEMail) that we created in Step 2. You see that we also pass some of the Task arguments which are used by the PowerShell script to customize the email with the exact error condition associated with the Alert:

Once the Data Collector is configured properly you will want to start it.

If you configured everything correctly you should start receiving emails any time that alert threshold is surpassed. I also manually ran the script once from PowerShell with no Argument Data just to make sure the script would generate the email to my E-Mail Address.

Step 4 – Set the Data Collector Set to run upon server startup.

You now have one more step. Whenever you reboot a server the Perfmon Counter Alert will not start automatically. In order to survive a reboot you must run the following at a command prompt.

Note: “Processor Usage Alert” referenced in the script below is the name of my user defined Data Collector Set.

I now have a simple way to monitor my Web Server should this event happen again with the processor being pegged over a two minute period.

PLEASE COMMENT! I LOVE IMPROVEMENTS TO MY SCRIPTS! HAPPY TROUBLESHOOTING!

Reference:
Setup E-Mail Generation from a Performance Counter Alert

Killer Deal for a Standard Single Domain SSL Certificate – GoDaddy

I was in the market for renewing my VPN appliance SSL certificate and came across this great deal that I wanted to pass along. It gives you a new SSL Certificate for a single domain with basic validation for only $5.99 for the first year. I usually get a new certificate for my VPN appliance yearly so this works out great!

Promo Code for GoDaddy SSL Purchase

LINK HERE: https://www.godaddy.com/web-security/ssl-certificate

Just enter that code above when you are at checkout in the Promo Code field and you will get the certificate for $5.99! Great Deal. I am not sure how long this will last, so if you try it and it does not work, let me have knowledge of that and I will take this post down! Just remember to always look for these hidden gems when purchasing web related items like domains and certificates. You can always find deals if you look hard enough.

HAPPY TROUBLESHOOTING! HAPPY SAVING!

References:
GoDaddy SSL Certificate Coupons
TechSmart Info on GoDaddy SSL Certificate Promotion

Check Windows Updates Installed via PowerShell

I had an issue last night where a server lost Secure Channel Connection to the PDC Emulator (NETLOGON Event IDs 5719 and 5783). All tests to test the secure channel via PowerShell were failing. (i.e. nltest or Test-ComputerSecureChannel cmdlets) The server essentially needed to be rebooted. I had a dumb dumb in my brain and forgot to check to see if there were any pending Windows Updates, because those need to be installed at the proper time and to a schedule. So, when I ran the following command to reboot:

The Windows Updates were installed inadvertently which could have caused even more issues if they were NOT approved or caused another failure on the server. TO NOT DO THIS IN THE FUTURE, remember to run the following command to shut off the Windows Update Service BEFORE initiating the reboot of the server:

But, the deed was done. NOW, I had to find out quickly what updates WERE installed via PowerShell so that I could alert the proper folks and give them a heads up on possible issues. Luckily, the server did NOT have any issues and the initial problem with NETLOGON was resolved. Here is the command I ran to find out the installed hotfixes filtered by today’s date:

Here was the Output:

Caption=http://support.microsoft.com/?kbid=4480960 
CSName=DC01
Description=Security Update 
FixComments= 
HotFixID=KB4480960 
InstallDate= 
InstalledBy=NT AUTHORITY\SYSTEM 
InstalledOn=2/26/2019 
Name= 
ServicePackInEffect= 
Status= 

Caption=http://support.microsoft.com/?kbid=4480965 
CSName=DC01 
Description=Security Update 
FixComments= 
HotFixID=KB4480965 
InstallDate= 
InstalledBy=NT AUTHORITY\SYSTEM 
InstalledOn=2/26/2019 
Name= 
ServicePackInEffect= 
Status=
 

Since there were no issues, I was able to resolve the incident. I did notify the account team though of the inadvertent installation so that they could revert the changes if necessary.

Remember, troubleshooting to resolution is a methodical process, and when in an enterprise environment, you MUST be aware of all factors of change process, even when the resolution is a simple reboot of the affected server.

HAPPY TROUBLESHOOTING!
I LOVE COMMENTS! THANKS FOR READING!

References:
Methods of generating installed updates via PowerShell
Check Windows Update History via PowerShell
Disable or Bypass Windows Update Installation During Reboot/Shutdown of a Server

Website Availability Checker Script

We get alerts from time to time that SharePoint and Skype Websites might not be loading correctly or not at all. The following is a script that will generate a report, with time and date, showing the availability of the websites in question and the time it takes to load them.

First off, you want to create a text file called URLList.txt that will have the list of websites you want to check. I save the file in the C:\PowerShell directory but you can change the path to where you need it. If this happens often, having this list prepared will help you evaluate the environment quickly and allow you to troubleshoot if necessary efficiently.

URL List Text File

Here is the script:

Here is your output:

PowerShell Output
The report shown in your web browser

HAPPY TROUBLESHOOTING!
MAKE THIS SITE BETTER!
PLEASE COMMENT!

Reconnecting Shared Mailboxes after an O365 Migration

I get a lot of these incidents in my queue after a user has been migrated to O365. For whatever reason, most likely due to the mailbox being moved itself, whether it is the user’s mailbox, the shared mailbox, or both, the connections to the shared mailboxes stop working in Outlook and the user cannot connect to the shared mailbox.

Here is a quick and easy solution to use to disconnect and reconnect the shared mailbox(es) that you lose connectivity to when migrated. This is usually performed on Outlook 2016 and above as most users upgrade their client software when moved to O365.

First, we remove the existing shared mailbox connection:

  • Click the File > Account Settings > Account Settings.
  • Select your company email address in the account list.
  • Click Change > More Settings > Advanced tab > Select the Shared Mailbox > Remove
  • Click Apply > OK > Next > Finish.
  • The shared mailbox will now automatically be removed in your Folder pane in Outlook.

Second, we re-add the shared mailbox connection to Outlook:

  • Click the File > Account Settings > Account Settings.
  • Select your company email address in the account list.
  • Click Change > More Settings > Advanced tab > Add
  • Type the name of the shared mailbox in the window and click OK.
  • Click Apply > OK > Next > Finish.
  • The shared mailbox will now automatically be added to your Folder List pane within Outlook.

Note: The above procedure must be followed in order to properly reconnect the shared mailbox. You cannot remove and re-add the mailbox in the same process as that will not reset the connection properly. You must save the settings when disconnecting.

I hope that this will assist everyone when troubleshooting Outlook connectivity issues to shared mailboxes after a migration.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!

Moving mailboxes to O365 via PowerShell in Hybrid Configuration

As many of you have knowledge, I am studying for my MS-202 Exam. And, part of the knowledge needed is to be able to migrate mailboxes between on premises and Exchange Online through PowerShell. Here are the steps for the scenario to move a mailbox from on premises to O365:

1. Connect to Exchange Online via PowerShell

If you have read my previous post: Connect to All PowerShell Modules in O365 with one script
You should have all the settings needed to connect your PowerShell to O365. Note in this scenario, that all these cmdlets will be run from O365 PowerShell and will be monitored from O365 by either PowerShell or the Exchange Admin Center. You will not be able to monitor the moves from On-Premises.

2. Provide your on premises Migration Administrator credentials as a variable for your cmdlet.

3. Move a single mailbox.

In your hybrid configuration you should be doing directory sync with O365/Azure and the accounts should be available in the cloud showing that they are synced with AD. This also assumes that you have your MRS Proxy endpoint enabled, which can be done by the HCW. Also, make sure you have your licensing available for your mailboxes. From my knowledge, you can assign your license to the account in the cloud before moving, especially if you have a particular license that you need to assign the account. Other than that, moving the mailbox will assign an existing license that is available that includes an Exchange Online mailbox feature when the mailbox is moved.
Now we initiate the move with the cmdlet. Similar to what you would do in the GUI, this simple mailbox move cmdlet initiates the move request. It has most of the same parameters as a local move request including BadItemLimit, LargeItemLimit, AcceptLargeDataLoss, etc…
 
Use the following LINK for documentation on the New-MoveRequest cmdlet.

Now with all migration projects, we expect to have to move multiple mailboxes in a single batch. The following will show the process for moving mailboxes in bulk from on premises to O365:

1. Connect to Exchange Online via PowerShell

If you have read my previous post: Connect to All PowerShell Modules in O365 with one script
You should have all the settings needed to connect your PowerShell to O365. Note in this scenario, that all these cmdlets will be run from O365 PowerShell and will be monitored from O365 by either PowerShell or the Exchange Admin Center. You will not be able to monitor the moves from On-Premises.

2. Provide your on premises Migration Administrator credentials as a variable for your cmdlet.

3. Move multiple mailboxes in a single batch.

In your hybrid configuration you should be doing directory sync with O365/Azure and the accounts should be available in the cloud showing that they are synced with AD. This also assumes that you have your MRS Proxy endpoint enabled, which can be done by the HCW. Also, make sure you have your licensing available for your mailboxes. From my knowledge, you can assign your license to the account in the cloud before moving, especially if you have a particular license that you need to assign the account. Other than that, moving the mailbox will assign an existing license that is available that includes an Exchange Online mailbox feature when the mailbox is moved.

This time you want to create a CSV file using the alias or emailaddress as your header and then list the appropriate value for all the users in your batch group. Save the file locally as MigrationBatch01.csv or a name of your choice.

Use EMailAddress
 OR
 Alias as the header

Next you initiate the mailbox moves. When specifying the mailbox identity in the cmdlet, use the respective header in your variable declaration (either $user.EMailAddress OR $user.Alias)

Use the following LINK for documentation on the New-MoveRequest cmdlet.

GOOD LUCK WITH YOUR MIGRATIONS!
HAPPY TROUBLESHOOTING!

References:
Moving Individual Mailboxes to O365
Move Mailboxes in Bulk to O365
PowerShell Mailbox Migration to O365
Connect to all PowerShell Modules in O365 with one script
New-MoveRequest Microsoft Document

Exchange DAG Replication Problem: An established connection was aborted by the software in your host machine

I had an issue with a four node DAG where the DR site with two of the DAG members were having replication issues. It was only technically affecting one DAG Member though. The copy queue length was really high and the logs were not committing to the database. A Test-ReplicationHealth cmdlet test told that the copy queue length for the affected database copy was high. No other databases were affected as there were eight databases on this DAG Node. The issue was that the log files were not replicating properly to the one DAG member for that database, causing the log file drives on all the other DAG members to build and become full:

EX04 DAG member has high Copy Queue Length
The purple member (EX04) free space is different from the other three DAG members

Circular Logging was turned on, but since the db was NOT in sync, the logs could NOT truncate properly which rendered CL useless. What was being done to stave the issue was to suspend the database copy of the affected DAG member (EX04), then resume the copy. The logs would replay and commit to the database copy on the DAG member, but over a short period of time, the same issue would arise again, as shown in this graph:

You can see the other DAG members start dropping in free space

There were absolutely no errors in the Event Viewer showing this replication issue. After some research, I ran the following cmdlet showing a particular output parameter that gave me the actual problem:

Get-MailboxDatabaseCopyStatus DAG1DB01 | ft -a -wr Name, Status, IncomingLogCopyingNetwork

Output with the actual error listed for the DR DAG members.

The operative error here was: {An error occurred while communicating with server ‘EX01’. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.} 

Now even though only EX04 was actually having problems with its log replication, both DR members EX03 & EX04 were having the same problem. Again, there were NO events in event viewer showing this issue. I next did some connectivity tests to EX01 from EX04 even though the error said there was an established connection that was broken.

Ping EX01 -f -l 1472

Now the -f states do NOT fragment the packet and send it as a whole to the destination.
The -l states the packet/buffer size you want sent. In this case 1472 bits.
By doing this, you are able to assure that a router or switch is NOT segmenting the packets, packet segmentation of replication logs can cause data corruption and replication issues.

That test passed successfully. I also did a trace route to assure there was no packet loss on the route to the replicating server. That test passed successfully.

I next checked the DAG Network to assure that all networks were working for replication. Now, in this scenario, there was only ONE DAG Network, there was NOT a separate Replication Network. I did not design the DAG and limitations most likely came into play during the design. From my experience, you setup a separate replication network for replication only, but if your network has enough bandwidth, and the design calls for simplification, you can use one DAG network in your design.

Get-DatabaseAvailabilityGroupNetwork | fl 

RunspaceId : a1600003-8074-4000-9150-c7800000207f 
Name : MapiDagNetwork 
Description : 
Subnets : {{192.168.1.0/24,Up}, {192.168.2.0/24,Up}} 
Interfaces : {{EX01,Up,192.168.1.25}, {EX02,Up,192.168.1.26},{EX03,Up,192.168.2.25}, {EX04,Up,192.168.2.26}} 
MapiAccessEnabled : True 
ReplicationEnabled : True 
IgnoreNetwork : False 
Identity : DAG1\MapiDagNetwork 
IsValid : True 
ObjectState : New 

All the DAG Network Members were up and not showing errors. I next did a telnet session to EX01 over the default DAG replication port 64327 to see if there would be any connectivity issues to EX01:

telnet EX01 64327

That test was successful and there were no connectivity issues to EX01 from EX04. Again, there was only ONE database out of eight that was having replication problems. After mulling over the problem, it was decided to restart the MSExchangeRepl service on EX03 AND EX04 since the error was present on both DAG members. We would then, suspend the database copy and resume the database copy on the affected servers.

Run on EX03:
Restart-Service MSExchangeRepl
Suspend-MailboxDatabaseCopy DAG1DB01/EX03 -Confirm:$False
Resume-MailboxDatabaseCopy DAG1DB01/EX03 -Confirm:$False

Run on EX04:
Restart-Service MSExchangeRepl
Suspend-MailboxDatabaseCopy DAG1DB01/EX04 -Confirm:$False
Resume-MailboxDatabaseCopy DAG1DB01/EX04 -Confirm:$False

After monitoring the databases and log drives, the issue was resolved and replication started functioning properly.

Log Drive Available Space Returned to Normal for DAG members

PLEASE COMMENT! I WELCOME SUGGESTIONS, TIPS, ALTERNATIVE TROUBLESHOOTING! HAVE A GREAT DAY!

Customize your Default PowerShell CLI Prompt

We all like to have our customization in our Windows Desktop. Custom colors, icons, wallpaper, etc.. Well IT guy/gal, why not have your PowerShell CLI the same way? I’ve looked around at a few blogs and got some ideas to share with you on customizing your PowerShell CLI.

Now, by default, Windows looks in the following directory for your customization file:

C:\Users\(username)\Documents\WindowsPowerShell

It looks for a file called profile.ps1 and will load that script every time you load PowerShell once it is customized.

You can construct the script within PowerShell ISE or your favorite editor. The following is how I programmed the script to customize my PowerShell prompt:

First, we want to clear the slate on the PowerShell CLI. 
I like to add my contact information for instance.

Second, I run a script that I came across and added to my customization.
It get’s me the weather for the local area that I am from. Click the link for details.

How To Uniquify Your PowerShell Console (Scroll to Getting the Weather)

Next, I customize the PowerShell Window Settings so that it is the size and shape that I want.
I set the Directory Location, The Colors, and The Window Sizes.

Next, I wanted to write some text in the window before I get my PowerShell Prompt.
I work at Avanade, so I wanted to put a welcome message, today’s date, and what PS version I was running just because I could.

Lastly, we actually configure the prompt. There is a lot of ways to do this and I will leave references for you at the bottom of this post so that you can get more details on the commands actually run.
I wanted to have a colorful prompt that stated the company motto and put the current time.
I also configured the Window Title at the top to show text, the current user, and the current directory with different colors as that is my expressive part of my brain. 🙂

Here is the script in its entirety:

Here is the final product when opening PowerShell:

Avanade Custom PowerShell Prompt
My Customized PS Prompt

HAPPY SCRIPTING!
HAPPY VALENTINES DAY!

References:
Customizing your PowerShell Profile
Get-Weather.ps1
Modify your PowerShell Prompt
PowerShell Basics: Console Configuration

Veeam Backup Validation Tool

I ran across an issue with my VM backups saying that they were failing validation and not backing up properly, even though each VM showed success when checking the logs. I was getting a specific error in the backup logs:

Backup files health check has been completed
Failed to perform backup file verification Error: Data error (cyclic redundancy check). Failed to read data from the file [B:\Backups\LDLNET Other Backup\LDLNET BackupD2019-01-12T001234.vbk]. Agent failed to process method {Signature.FullRecheckBackup}.

So, I did some research and found a little known tool that is used to manually validate the Veeam backup files, basically because it’s a tool usually executed only by the technical support staff. It is located in the following folder (Version 9.5.0.1922):

C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Validator.exe

Its main use case is to verify the consistency of a backup created with Veeam Backup & Replication. It’s NOT SureBackup, that does another kind of control by starting the VM from the backup file, and is for sure more reliable. But if you do not want to start a SureBackup activity, or if you only have a Standard license lacking SureBackup, this tool can be a good alternative, or you can use it to check a backup file after it has been moved or if you had a consistency problem on the storage holding those files.

The command switches are listed here for the executable

Now since I had a specific file that was showing an error, I wanted to run the command against that file to validate the backup. Here is the command that I ran:

.\Veeam.Backup.Validator.exe /file:”B:\Backups\LDLNET Other Backup\LDLNET BackupD2019-01-12T001234.vbk”

It started running against the file and it did fail as it did in the log files from the backup process. Here is the error message:

Skipping VM ‘B:\Backups\LDLNET Other Backup\LDLNET BackupD2019-01-12T001234.vbk ‘: File “LDLNET-VM01-flat.vmdk” is corrupted. Data error (cyclic redundancy check).
Failed to read data from the file [B:\Backups\LDLNET Other Backup\LDLNET BackupD2019-01-12T001234.vbk ].

Now, when looking at the backup job, I found that the file listed was the original full backup that I had completed when I originally changed the job for all the new VMs that were now listed in the backup job. Since that was the case and I did not have any VMs that were in a bad state, I deleted all the backup files from my storage and started another full backup of the VMs in the job.

Using the CLI tool to manually validate the backup file was very helpful in this case as it would help me decide to clear out a backup that would not restore properly, even with the incremental backups since the base full file was corrupt.

References:
Veeam Backup Validator: check the consistency of your backup files

Connect to all PowerShell Modules in O365 with one script

Let’s say you’re an admin that needs to connect to Office365 via PowerShell often. Now, there are many different websites or blogs that will show you how to connect to each session via PowerShell. That can cause a headache since you can end up having five different PowerShell sessions running in five different windows. You end up having to enter a username and password all those times, which can become time consuming.

I want to show you here how to combine all those sessions into one script where, if you’re security is tight enough on your computer, you don’t even have to enter credentials. This way, you can click on one icon and pull up all the O365 PowerShell commands that you’ll need to manage your organization.

First you need to download the following PowerShell Module Installation Files so that your PowerShell Database will have the correct modules installed:

Microsoft Online Service Sign-in Assistant for IT Professionals RTW
Windows Azure Active Directory Module for Windows PowerShell v2
SharePoint Online Management Shell
Skype for Business Online, Windows PowerShell Module

Next, we want to setup the CLI (Command Line Interface) to be too cool for school. I have learned it helps to have knowledge of how to customize the CLI window. You can do all of this in PowerShell ISE or Notepad, which ever you prefer. Here are the commands for the script that I use to setup the CLI:

Next, you want to set your Execution Policy and put in your credentials so that you won’t be prompted to enter the user credentials when you run the script.

NOTE: MAKE SURE YOU KEEP YOUR SCRIPT SAFE AS THE CREDENTIALS ARE VISIBLE WITHIN THE SCRIPT IN PLAIN TEXT!

You can, alternatively, set your script to prompt for credentials every time by using the following:

$LiveCred = Get-Credential

Here is that part of the script:

Now we get into the importing of the modules for each O365 service:

Get the MSOnline Module:

Connect to the MSOnline Service:

Connect to Azure AD PowerShell:

Connect to SharePoint Online PowerShell:
NOTE – MAKE SURE YOU CHANGE TO YOUR COMPANY NAME IN THE URL!!

Connect to Exchange Online PowerShell:

Connect to Skype For Business Online PowerShell:

Connect to the Security & Compliance PowerShell:
NOTE – This one I still get “Access Denied” when trying to connect. I have looked for an answer to that issue, but have not found one. Please comment with a link if you have an answer so that I can update this script!

Lastly, put in a note to show that the PS load is completed:

So Here is the final script in its entirety:

Now you can create your icon for your desktop so that you can easily access the script. I would save the script to your Scripts directory.

That will usually be C:\Users\’username’\Documents\WindowsPowerShell\Scripts or wherever directory you choose.

To start, right click the desktop and choose New > Shortcut
In the Target Field, enter the following for your PowerShell Shortcut, pointing to the path of your script:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -ExecutionPolicy Unrestricted -File “C:\Users\username\Documents\WindowsPowerShell\Scripts\ConnectO365All.ps1”

Click on the Advanced button and check the box: Run As Administrator
Under the General Tab, name your shortcut: (CompanyName) O365 All PowerShell
Click OK to save the shortcut to your desktop.

LAST BUT NOT LEAST, RUN THE FOLLOWING COMMAND BEFORE EXITING OR CLOSING YOUR POWERSHELL WINDOW. THIS WILL REMOVE ALL THE SESSIONS YOU’VE CONNECTED TO:

Get-PSSession | Remove-PSSession

HAPPY SCRIPTING!
LEARN, DO, LIVE!

References:
Connect to all O365 Services in one PowerShell Window
How to connect to all O365 Services through PowerShell
Connecting to Office 365 “Everything” via PowerShell

Checking Drive Space Volumes for DAG DB members through PowerShell

I had received a weird alert for a DB volume for a DAG member being below threshold. This was odd to me due to the fact that there were four DAG members and we only received an alert for one. I went into Azure Log Analytics and ran the following query to render a graph for the past 14 days showing the percent free space of the volume for all the DAG members.

Thanks Georges Moua for the query script!

Now the reason I can run the query this way is due to the fact that the Design of the DAG was correctly done and the DB folders are identical on all DAG members. The query rendered the following chart:

As you can see the Green DAG member is way below the other DAG members.

I next went to an Exchange Server in the DAG and got the volume data for all the members in the DAG:

EX02’s volume free space is far below the other DAG members

I went on EX02 and found that there was a subfolder named “Restore” that was not present on the other servers. I ran the following script to get the size of that folder in GB:

The folder size was 185 GB. Removing that folder, along with all subfolders/files, would balance the free space to the other DAG members. I ran the following cmdlet to remove the folder and all subfolders/files:

This remediated the alert and balanced the drive space across all DAG members.

POST YOUR COMMENTS OR QUESTIONS!
HAPPY TROUBLESHOOTING!

Event 11022 with MSExchangeTransport – Easy Validation Test

In a hybrid environment, you’re always connecting between the cloud and on premises to establish transport through the connectors to transport mail. By default, this is done over a TLS (Transport Layer Security) connection. It’s similar to a VPN or SSL connection using certificates on the Transport Layer of the network stack to encrypt the data between the two Organizations in a Hybrid configuration.

Because you are using certificates, the certificate must be validated properly and checked to see if it has expired or been revoked by the issuing company. A revocation list is created and updated regularly for this purpose. If the connecting organization cannot validate the revocation of the certificate, it will not establish a TLS connection with the connecting organization. You will then get the following event:

Event 11022
MSExchangeTransport
Error:
Failed to confirm domain capabilities ‘mail.protection.outlook.com:AcceptOorgProtocol’ on connector ‘Inbound from Office 365’ because validation of the Transport Layer Security (TLS) certificate failed with status ‘RevocationOffline’. Contact the administrator of ‘mail.protection.outlook.com’ to resolve the problem, or remove the domain from the TlsDomainCapabilities list of the Receive connector.

Most likely, there is a network issue with the On Premises Organization being able to retrieve the Revocation File with the Certificate Information. Since it cannot retrieve that file, it stops the transport connection and throws the error.

A simple validation to validate the connector and assure transport from Office365 is to run the following cmdlet from the server on premises that performs the connection:

Again, I like to put the other cmdlets of 
write-host, hostname, and date 
in order to make it easy to document when working an incident.

From the highlighted text, we can see the test was successful.

The test runs a connection for each connector and tests the validity of each connector. If a success is returned, then we have knowledge that the certificate was validated and the connection was established through the connector from Office365.

If you get a failure though, you will need to run tests to see if you can pull the revocation list for the certificate as well as a simple test to connect to Office365:

Connect to Exchange Online via Powershell

IMPORTANT NOTE

I wanted to put some information on how to pull the CRL Distribution Point for the Office365 so that you could run an Invoke-WebRequest to pull the CRL file from the Distribution Point, but I have NOT found a single way through Powershell to pull that information. I have searched multiple posts and articles showing all these advanced methods of using certutil and PowerShell to get a bunch of other information, but NOTHING on how to pull the URL for the CRL file from the certificate. Doing a Get-ChildItem for the certificate using the Thumbprint does NOT pull that property from the certificate. Now, if you have a cmdlet that WILL do that, PLEASE POST!

So, in essence, to troubleshoot if you can get to the CRL file, you get the URL for the CRL Distribution Point from the GUI Properties of the certificate. Then you run the following cmdlet in PowerShell:

POST COMMENTS!
HAPPY TROUBLESHOOTING!

What the Hybrid Configuration Wizard Performs in the background and configuring Hybrid Co-Existence with Exchange Online

I’m working on getting certified in Exchange Hybrid Scenarios and Exchange Online configuration as part of my skill set for Exchange. In doing so, I had successfully implemented a complete Full Hybrid Exchange Environment between my Exchange Online Tenant and my On Premises Exchange 2019 Environment last evening.

I wanted to give an update that was posted to my LinkedIn Posting on this. Thank you Brian Day for the vote of confidence and caution that running these cmdlets manually is not supported by Microsoft and that the HCW, like all the Online Microsoft Products, is constantly changing and being updated.

Important Note

As preparation, I bought some Exchange Online Plan 1 licenses which give me a 50 GB mailbox limit and basic mailbox functionality. It does not include the more advanced features such as ATP, or DLP. I am running most of those features through my On Premises Environment. I mainly wanted to be able to place mailboxes in the cloud and have a hybrid setup. My plan was to have mail flow continue through my On Premises environment so that my Exchange Server features would be used and I would not have to change any MX or SPF records. I also had my certificates in place for SSL and OWA so I would want keep mail flow routed that way, through on premises. I do want to be able to have Free/Busy lookups cross-premise so federation would have to be enabled as well. I would also have to enable the MRS proxy on my Exchange Server so that mailbox migration could be implemented cross-premise. I also have previously configured Azure AD Sync along with ADFS for Single Sign On. In my case, another server was not needed as I didn’t have enough mailboxes or real need to split my frontend and backend deployment. Running the Hybrid Configuration Wizard would not open any new ports or change any existing port traffic that was already configured on my firewall. These are just a few of the considerations that need to be looked at when considering a hybrid integration.

Here is a great article to read for the prerequisites
Exchange Hybrid Deployment Pre-requisites

So, once I had all those considerations handled in my design, I ran the Hybrid Configuration Wizard. What I want to do in this blog post is to go through the steps that the wizard does in the background to setup the Hybrid Environment as you go through the Wizard.

I mainly used the following blog post as a reference, but have approached it differently by diving into the cmdlets that are run during the process:

1. The HCW validates the On-premises and Online Exchange Connection.

The Hybrid Configuration Wizard checks if it is possible to connect to both servers with PowerShell. It runs the Get-ExchangeServer cmdlet on premises after resolving the server in DNS. It then connects to Exchange Online, authorizing the connection:

Authority=https://login.windows.net/common Resource=https://outlook.office365.com ClientId=abcdefgh-a123-4566-9abc-2bdflancelin

2. The HCW collects data about Exchange configuration from the on-premises Active Directory

The Wizard gathers information about the local domain. In order to do that, the HCW executes a series of cmdlets.

These include, in order:

3. The HCW collects information on the Exchange online (Office 365) configuration

This task repeats what has been done in the previous step, only for the Exchange online, instead of the on-premises one.

The cmdlets include, in order:

4. Federation Trust is determined. If not present, a new Federation Trust and the required certificate will be created on the local Exchange Server

You will be prompted in the Wizard to create a Federation Trust if not present. The following articles explain Federation and its requirements:

Understanding Federation – Link Here
Understanding Federated Delegation – 
Link Here
Create a Federation Trust – 
Link Here

If the activity is finished successfully, a new certificate should appear on the on-premises Exchange Certificates list. The new certificate includes “Federation” in its Subject field. To make sure the certificate is there, you can run a cmdlet: Get-ExchangeCertificate | ft -a -wr


The results will look like this

5. The HCW creates a new Hybrid Configuration Object in the local Active Directory

The HCW will run cmdlets based on the information you provide in the HCW for the certificate, the on premises Exchange Server, the domain(s), and what features you want turned on:

It then checks the settings through the following cmdlets:

It then enables Organization Customization for both environments through this cmdlet:

6. Configuration is then completed to modify the settings on the on premises Exchange environment 

EmailAddressPolicy – HCW adds address @tenant.mail.onmicrosoft.com
The HCW configures remote domains – adds tenant.mail.onmicrosoft.com and tenant.onmicrosoft.com
The HCW adds a new accepted domain – adds tenant.mail.onmicrosoft.com

Some of the cmdlets run:

7. The HCW Configures the Organization Relationship between the local server and the cloud.

This configuration is not necessary in minimal hybrid deployment. Since I have a full hybrid deployment configured, the cmdlets were run as needed to configure it. Thanks to the correct configuration, it is possible to synchronize free/busy status of mailboxes and their elements between the on-premises Exchange Environment and Exchange online. 

Some of the cmdlets run in the process:

8. The HCW and setting connectors on both Exchange servers

The HCW checks to see if the connectors are there, if not, it sets them up. During this workflow, four connectors are set – one receive and one send connector for each server. Those connectors guarantee the mail flow between the on-premises and Exchange Online.

Some of the cmdlets run in the process:

The Intra-Organization is set as well:

9. The HCW configures OAuth Authentication across the Hybrid

This LINK explains how OAuth is configured between Exchange On Premises and Exchange Online. It’s a very good article to read as it shows how to get the Modern Authentication style working. Now the HCW does this for you and at the end of the article, you can run cmdlets to test the validity of the configuration.

If you want to go into a deep dive about how the Hybrid Authentication works, see the following:
Deep Dive Into Hybrid Authentication – from the MS Exchange Team Blog

Here are some of cmdlets run during this process workflow:

Again, look at both of those links to get a little more detail as to what each cmdlet does and how it sets up OAuth. Here are the two cmdlets used to test OAuth:

10. Enable MRS Proxy for Migration

In order to be able to move mailboxes between Exchange On Premises and Exchange Online, you have to enable the Exchange Web Services Virtual Directory to use the MRSProxy (Microsoft Replication Service proxy). You also have to set your EWS Virtual Directory to use Basic Authentication. You’ll want to do this before running the HCW or else you will receive the following error when the HCW validates the Migration setup and configuration:

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server ‘mail.ldlnet.net’ could not be completed. —> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to ‘https://mail.ldlnet.net/EWS/mrsproxy.svc’ failed. Error details: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. –> The remote server returned an error: (403) Forbidden.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned an error: (403) Forbidden.

Some of the cmdlets run to test Migration and MRS Proxy Settings are as follows:

11. Final HCW Configuration and cleanup.

The HCW runs from final cmdlets to finish up the installation of the Hybrid environment. Here are the cmdlets run:

All this information was found in the setup logs that are in the following directory
C:\Users\%username%\AppData\Roaming\Microsoft\Exchange Hybrid Configuration

REFERENCES
Understanding Federation
Understanding Federated Delegation
Create a Federation Trust
Hybrid deployment prerequisites
Exchange Specific OAuth 2.0 Protocol Specification
Understanding WS-Security
JSON Web Tokens
Using OAuth2 to access Calendar, Contact and Mail API in Office 365 Exchange Online
Configurable token lifetimes in Azure Active Directory (Public Preview)
OAuth Troubleshooting
Principles of Token Validation
Troubleshooting free/busy issues in Exchange hybrid environment
How to configure Exchange Server on-premises to use Hybrid Modern Authentication
Microsoft 365 Messaging Administrator Certification Transition (beta)
Microsoft 365 certification exams
Exchange Server build numbers and release dates

PLEASE LEAVE QUESTIONS, COMMENTS, UPDATES! I WOULD LOVE TO HEAR FROM YOU!

The Complete Guide To PowerShell Punctuation

In this short little snippet, I wanted to provide a good download that I had found. In my searching for more knowledge through PowerShell, I found this great PDF file that has listed all the different symbols used in PowerShell Scripting and their meanings.

Here is the download:

Example of the available file.

Please feel free to download it and use it in your education of PowerShell. I have knowledge that I will be using it. The website I got it from has some good explanation of functions and punctuation for PowerShell as well. See below…

Reference:
PowerShell Punctuation

Measuring CPU Processor Times Per Core Across Multiple Servers through PowerShell.

I want to thank Jason Field for the bulk of this script!

Our team was presented with an issue where we needed to measure the CPU Percentage Processor Times for Each Core within the Physical Processor and be able to output that data quickly through PowerShell. We all have knowledge that Performance Monitor can do this through a GUI, but it is very difficult to be able to output that data to a file in a manner that can be easily read. We had an original PowerShell cmdlet that would accomplish this for the total percent processor time for all cores over a one minute period:

Our challenge was to be able to do this per core, over the same time period, and get the average for each core, so that we could measure the output accordingly. I had been working on a script that was able to run the command, but not in a parallel fashion. The script was running in sequence and was taking way longer than one minute to complete. Frustrating to say the least.

In comes Jason Field, showing me the meaning and value of the back tick as well has how to run and monitor job functions in a PowerShell Script so that the task could be completed, as needed, across a server array.

The main purpose of the (back tick) is that it allows for variables to be used in the script block on the remote server instead of being filled in before creating the script block.

Here is the script:

Sample Output from the Script.

What I learned from this script is that the back tick ” ` “ allows for multiple commands to be run in a sub-routine within the script job and be gathered before the main script command is run and the output given. This gets past the multi-threading issue I was having with my original script. The script can then be run across multiple servers using the Invoke-Command cmdlet or over a remote PowerShell session as a Job. The jobs can then be monitored as the scripts finish across the multiple servers in the time period given for the samples. I had modified the script to do the multiple samples and then take the average of the CookedValue per the original cmdlet. I could not, however, get the ExpandProperty parameter to work with the script.

Please, as I am still learning, if you see an error with this script, please alert me with a comment or contact me directly so that I can update the script properly. 

Thanks again Jason! It really was like a light bulb going “DING” when I figured it out with concern to the back tick. I had also been having issues with the active job monitoring process. It was a real help!

PLEASE COMMENT, SHARE, AND HAPPY TROUBLESHOOTING!

Update Edge Server Certificate in a Hybrid Exchange Environment

LDLNET LLC Banner
LDLNET LLC – Life In Action! Your Source for Professional IT Services!

At work, our group was updating the Exchange Edge Server certificates and having mail flow problems causing messages to be in the Poison Queue and not transfer to Office365 properly. We finally got the procedure down to where it started working. I wanted to post that procedure here since I had never really worked with Edge Servers in the past. If this post can help you in the future, then “I done good!”

Now, everywhere I had read said that you have to remove and then re-create the Edge Subscription between your Transport Servers and the Edge Servers when changing the certificate.

Here is why:
When we subscribe the edge server, an AD LDS account called the EdgeSync Bootstrap Replication Account (ESBRA) is created. This is created using the default certificate private key of the certificate assigned to SMTP service as default, hence as long as we have that certificate the transport servers will be able to authenticate to the Edge server and replicate the required information to ADAM database.

Now when we install a third party certificate we assign SMTP service to it and overwrite the current certificate, basically we change the default SMTP certificate. So, by doing this, the current Edge Subscription will fail as the Edge server will not be able to decrypt the ESRA account passed on when communicating with the transport servers using the new certificate private key.

So, once you have your new 3rd party certificate, you install it to your edge servers:

Then, you enable the Exchange Certificate to be used for SMTP:

Mail flow will be broken at this point. Since messages were going to the poison queue due to the ESBRA account encryption failing when authenticating with the internal Transport Servers, I had to completely stop transport by disabling the Send Connectors between the internal Transport Servers and the Edge servers from the Transport Server.

The configuration of the Edge Servers were that there were two servers in the Edge Farm. Since one of the servers had not had a proper sync in a while, I decided to remove the recipient database that had been replicated to the failing server when removing the Edge Subscription. The other server, I left the recipient database in place so that we could get one server up and running quickly since transport was stopped at this point.

Here is the command that was run to remove the Edge Subscriptions. This needed to be completed on both the Edge Servers and the corresponding Transport Server:

I then had to create a new Edge Subscription file on each Edge Server to copy to the Transport Server. I already had connectors set so I did not need to recreate those connectors.

I copied the xml files of each Edge Server to the Transport Server and ran the following cmdlet to create the Edge Subscription to the Edge Servers. I then had the Edge Servers Rebooted for good measure before redoing a Full Manual Edge Sync.

I next had to preform a full manual EdgeSync from the transport server to the Edge Servers to assure that the recipient database on the AD LDS instance was up to date and that the send connectors were replicated properly.

I next had to re-run the Hybrid Configuration Wizard so that I could configure the Edge Servers as the transport for Hybrid cloud-bound Messages. Once the Edge Servers were chosen to transport Hybrid cloud-bound messages, I selected the new Edge Certificate so that transport would work properly when re-enabled and O365 would recognize the new certificate for Hybrid messages bound for the cloud.

I next re-enabled the Edge Send Connectors so that mail flow would begin working once the Full Edge Synchronization was completed. You have to let that complete before you can begin mail flow again so that messages won’t be delivered to the Poison Queue.

Mail flow began working. It took about 90 minutes for all the queues to clear properly that had queued messages waiting to transport. Any Poison Queued messages were removed with NDRs sent to the senders.

It was a doozy to say the least. Happy Troubleshooting!
Leave Comments or Questions you may have!

References:
Exchange 2010 Edge Transport Server: Configuring EdgeSync
Mail flow breaks after renewing SSL Certificate on Edge server with Edge Subscription
Start-EdgeSynchronization