Certificate – IT Blog

Lance LingerfeltFebruary 5, 2019

Event 11022 with MSExchangeTransport – Easy Validation Test

In a hybrid environment, you’re always connecting between the cloud and on premises to establish transport through the connectors to transport mail. By default, this is done over a TLS (Transport Layer Security) connection. It’s similar to a VPN or SSL connection using certificates on the Transport Layer of the network stack to encrypt the data between the two Organizations in a Hybrid configuration.

Because you are using certificates, the certificate must be validated properly and checked to see if it has expired or been revoked by the issuing company. A revocation list is created and updated regularly for this purpose. If the connecting organization cannot validate the revocation of the certificate, it will not establish a TLS connection with the connecting organization. You will then get the following event:

Event 11022
MSExchangeTransport
Error:
Failed to confirm domain capabilities ‘mail.protection.outlook.com:AcceptOorgProtocol’ on connector ‘Inbound from Office 365’ because validation of the Transport Layer Security (TLS) certificate failed with status ‘RevocationOffline’. Contact the administrator of ‘mail.protection.outlook.com’ to resolve the problem, or remove the domain from the TlsDomainCapabilities list of the Receive connector.

Most likely, there is a network issue with the On Premises Organization being able to retrieve the Revocation File with the Certificate Information. Since it cannot retrieve that file, it stops the transport connection and throws the error.

A simple validation to validate the connector and assure transport from Office365 is to run the following cmdlet from the server on premises that performs the connection:

write-host;hostname;date;write-host;Test-SmtpConnectivity | ft -a -wr | Out-String -Stream

1	write-host;hostname;date;write-host;Test-SmtpConnectivity \| ft -a -wr \| Out-String -Stream

Again, I like to put the other cmdlets of
write-host, hostname, and date
in order to make it easy to document when working an incident.

From the highlighted text, we can see the test was successful.

The test runs a connection for each connector and tests the validity of each connector. If a success is returned, then we have knowledge that the certificate was validated and the connection was established through the connector from Office365.

If you get a failure though, you will need to run tests to see if you can pull the revocation list for the certificate as well as a simple test to connect to Office365:

Connect to Exchange Online via Powershell

IMPORTANT NOTE

I wanted to put some information on how to pull the CRL Distribution Point for the Office365 so that you could run an Invoke-WebRequest to pull the CRL file from the Distribution Point, but I have NOT found a single way through Powershell to pull that information. I have searched multiple posts and articles showing all these advanced methods of using certutil and PowerShell to get a bunch of other information, but NOTHING on how to pull the URL for the CRL file from the certificate. Doing a Get-ChildItem for the certificate using the Thumbprint does NOT pull that property from the certificate. Now, if you have a cmdlet that WILL do that, PLEASE POST!

So, in essence, to troubleshoot if you can get to the CRL file, you get the URL for the CRL Distribution Point from the GUI Properties of the certificate. Then you run the following cmdlet in PowerShell:

Let's say the CRL file is at the following URL: http://crl.comodoca.com/COMODORSAOrganizationValidationSecureServerCA.crl 

Run the following:
Invoke-WebRequest http://crl.comodoca.com/COMODORSAOrganizationValidationSecureServerCA.crl -UseDefaultCredentials | Select * -ExpandProperty Headers | ft -a -wr

If you get a valid response from the output, then you're good!

Let's say the CRL file is at the following URL: http://crl.comodoca.com/COMODORSAOrganizationValidationSecureServerCA.crl

Run the following:

Invoke-WebRequest http://crl.comodoca.com/COMODORSAOrganizationValidationSecureServerCA.crl -UseDefaultCredentials | Select * -ExpandProperty Headers | ft -a -wr

If you get a valid response from the output, then you're good!

POST COMMENTS!
HAPPY TROUBLESHOOTING!

Lance LingerfeltJanuary 25, 2019April 19, 2019

Update Edge Server Certificate in a Hybrid Exchange Environment

LDLNET LLC Banner — LDLNET LLC – Life In Action! Your Source for Professional IT Services!

At work, our group was updating the Exchange Edge Server certificates and having mail flow problems causing messages to be in the Poison Queue and not transfer to Office365 properly. We finally got the procedure down to where it started working. I wanted to post that procedure here since I had never really worked with Edge Servers in the past. If this post can help you in the future, then “I done good!”

Now, everywhere I had read said that you have to remove and then re-create the Edge Subscription between your Transport Servers and the Edge Servers when changing the certificate.

Here is why:
When we subscribe the edge server, an AD LDS account called the EdgeSync Bootstrap Replication Account (ESBRA) is created. This is created using the default certificate private key of the certificate assigned to SMTP service as default, hence as long as we have that certificate the transport servers will be able to authenticate to the Edge server and replicate the required information to ADAM database.

Now when we install a third party certificate we assign SMTP service to it and overwrite the current certificate, basically we change the default SMTP certificate. So, by doing this, the current Edge Subscription will fail as the Edge server will not be able to decrypt the ESRA account passed on when communicating with the transport servers using the new certificate private key.

So, once you have your new 3rd party certificate, you install it to your edge servers:

Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path C:\Certs\EDGE-NEW.cer -Encoding byte -ReadCount 0))

1	Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path C:\Certs\EDGE-NEW.cer -Encoding byte -ReadCount 0))

Then, you enable the Exchange Certificate to be used for SMTP:

Enable-ExchangeCertificate -Thumbprint "The Certificate Thumbprint" -Services SMTP

1	Enable-ExchangeCertificate -Thumbprint "The Certificate Thumbprint" -Services SMTP

Mail flow will be broken at this point. Since messages were going to the poison queue due to the ESBRA account encryption failing when authenticating with the internal Transport Servers, I had to completely stop transport by disabling the Send Connectors between the internal Transport Servers and the Edge servers from the Transport Server.

Get-SendConnector -identity "Edge To Internet" | Set-SendConnector -Enabled:$False
Get-SendConnector -identity "Inbound - Edge To Internal Site" | Set-SendConnector -Enabled:$False

1 2	Get-SendConnector -identity "Edge To Internet" \| Set-SendConnector -Enabled:$False Get-SendConnector -identity "Inbound - Edge To Internal Site" \| Set-SendConnector -Enabled:$False

The configuration of the Edge Servers were that there were two servers in the Edge Farm. Since one of the servers had not had a proper sync in a while, I decided to remove the recipient database that had been replicated to the failing server when removing the Edge Subscription. The other server, I left the recipient database in place so that we could get one server up and running quickly since transport was stopped at this point.

Here is the command that was run to remove the Edge Subscriptions. This needed to be completed on both the Edge Servers and the corresponding Transport Server:

Get-EdgeSubscription | Remove-EdgeSubscription

1	Get-EdgeSubscription \| Remove-EdgeSubscription

I then had to create a new Edge Subscription file on each Edge Server to copy to the Transport Server. I already had connectors set so I did not need to recreate those connectors.

New-EdgeSubscription -FileName "c:\EdgeSubscriptionFiles\Edge01Subscription.xml" -CreateInboundSendConnector $false -CreateInternetSendConnector $false -Site "YourADSite"

1	New-EdgeSubscription -FileName "c:\EdgeSubscriptionFiles\Edge01Subscription.xml" -CreateInboundSendConnector $false -CreateInternetSendConnector $false -Site "YourADSite"

New-EdgeSubscription -FileName "c:\EdgeSubscriptionFiles\Edge02Subscription.xml" -CreateInboundSendConnector $false -CreateInternetSendConnector $false -Site "YourADSite"

1	New-EdgeSubscription -FileName "c:\EdgeSubscriptionFiles\Edge02Subscription.xml" -CreateInboundSendConnector $false -CreateInternetSendConnector $false -Site "YourADSite"

I copied the xml files of each Edge Server to the Transport Server and ran the following cmdlet to create the Edge Subscription to the Edge Servers. I then had the Edge Servers Rebooted for good measure before redoing a Full Manual Edge Sync.

New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\XMLImport\Edge01Subscription.xml" -Encoding Byte -ReadCount 0)) -Site "YourADSite" -CreateInboundSendConnector $false -CreateInternetSendConnector $false

1	New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\XMLImport\Edge01Subscription.xml" -Encoding Byte -ReadCount 0)) -Site "YourADSite" -CreateInboundSendConnector $false -CreateInternetSendConnector $false

New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\XMLImport\Edge02Subscription.xml" -Encoding Byte -ReadCount 0)) -Site "YourADSite" -CreateInboundSendConnector $false -CreateInternetSendConnector $false

1	New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\XMLImport\Edge02Subscription.xml" -Encoding Byte -ReadCount 0)) -Site "YourADSite" -CreateInboundSendConnector $false -CreateInternetSendConnector $false

I next had to preform a full manual EdgeSync from the transport server to the Edge Servers to assure that the recipient database on the AD LDS instance was up to date and that the send connectors were replicated properly.

Start-EdgeSynchronization -TargetServer Edge01 -ForceFullSync | fl

1	Start-EdgeSynchronization -TargetServer Edge01 -ForceFullSync \| fl

Start-EdgeSynchronization -TargetServer Edge02 -ForceFullSync | fl

1	Start-EdgeSynchronization -TargetServer Edge02 -ForceFullSync \| fl

I next had to re-run the Hybrid Configuration Wizard so that I could configure the Edge Servers as the transport for Hybrid cloud-bound Messages. Once the Edge Servers were chosen to transport Hybrid cloud-bound messages, I selected the new Edge Certificate so that transport would work properly when re-enabled and O365 would recognize the new certificate for Hybrid messages bound for the cloud.

I next re-enabled the Edge Send Connectors so that mail flow would begin working once the Full Edge Synchronization was completed. You have to let that complete before you can begin mail flow again so that messages won’t be delivered to the Poison Queue.

Get-SendConnector -identity "Edge To Internet" | Set-SendConnector -Enabled:$True
Get-SendConnector -identity "Inbound - Edge To Internal Site" | Set-SendConnector -Enabled:$True

1 2	Get-SendConnector -identity "Edge To Internet" \| Set-SendConnector -Enabled:$True Get-SendConnector -identity "Inbound - Edge To Internal Site" \| Set-SendConnector -Enabled:$True

Mail flow began working. It took about 90 minutes for all the queues to clear properly that had queued messages waiting to transport. Any Poison Queued messages were removed with NDRs sent to the senders.

It was a doozy to say the least. Happy Troubleshooting!
Leave Comments or Questions you may have!

References:
Exchange 2010 Edge Transport Server: Configuring EdgeSync
Mail flow breaks after renewing SSL Certificate on Edge server with Edge Subscription
Start-EdgeSynchronization

Lance LingerfeltJanuary 3, 2019January 3, 2019

Getting Certificate Information in PowerShell

When you have certificates expiring, you need to be able to gather the information about the certificates so that you can prepare the renewal requests properly and get the certificate renewed. Now, Windows doesn’t have a native application that is readily available to look up certificate data. You have to open the MMC console and then add the proper Certificate Snap-In to gain access to the certificate store.

In dealing with this, I have found that PowerShell is a great method to be able to gather all of this data quickly and in a way where you can copy/paste the information that you need in order to generate your request properly for a new certificate or a renewal.

First off, you have to make sure that the PKI Module is installed on your system that you are running PowerShell on:

Download and install PowerShell PKI module from the PowerShell Gallery using PowerShell

Install-Module -Name PSPKI

1	Install-Module -Name PSPKI

Module Requirements

Windows PowerShell 3.0 or higher
.NET Framework 4.0 or higher

This module can run on any of the specified operating systems:

Windows Server 2008*/2008 R2*/2012*/2012 R2*/2016*/2019*
Windows Vista/7/8/8.1/10

* — Server Core installation is not supported.

NOTE: Module installation requires installed RSAT (Remote System Administration Tools)

Once you have it installed, you can then begin accessing the Certificate Store on the server that you are on:

Import-Module PKI

1	Import-Module PKI

Set-Location Cert:\LocalMachine\My

1	Set-Location Cert:\LocalMachine\My

NOTE: Setting the location to LocalMachine\My will place PowerShell in the Personal Store of the Local Computer Account.

The Get-ChildItem cmdlet will return the information of the certificates that are in the directory that you are in. You can also amend the cmdlet with given parameters to get the information from another machine:

To get the properties of all certificates expiring in 120 days locally:

Get-ChildItem –Recurse –ExpiringInDays 120 | fl -Property *

1	Get-ChildItem –Recurse –ExpiringInDays 120 \| fl -Property *

To get the properties of all certificates expiring in 120 days on a remote server:

Invoke-Command -ComputerName "ServerName" -ScriptBlock {Get-ChildItem -Path cert:\LocalMachine\My -Recurse -ExpiringInDays 120} | fl -Property *

1	Invoke-Command -ComputerName "ServerName" -ScriptBlock {Get-ChildItem -Path cert:\LocalMachine\My -Recurse -ExpiringInDays 120} \| fl -Property *

Now, let’s say that you have certificates expiring in 120 days on all of your CAS Exchange Servers and you need to get the information on all those certificates since they do not have the same thumbprint. You can run the following commands in sequence to be able to get the information from all of those servers:

$EXList = Get-ClientAccessServer | Select-Object Name | Sort-Object Name 

foreach ($EX in $EXList) { $EX.Name ; Invoke-Command -ComputerName $EX.Name -Command {get-childitem cert:\LocalMachine\My -Recurse | where-object {$_.NotAfter -gt (get-date)} | select *,@{Name="Expires in (Days)";Expression={($_.NotAfter).subtract([DateTime]::Now).days}} | where "Expires in (Days)" -lt 120 | fl} }

$EXList = Get-ClientAccessServer | Select-Object Name | Sort-Object Name

foreach ($EX in $EXList) { $EX.Name ; Invoke-Command -ComputerName $EX.Name -Command {get-childitem cert:\LocalMachine\My -Recurse | where-object {$_.NotAfter -gt (get-date)} | select *,@{Name="Expires in (Days)";Expression={($_.NotAfter).subtract([DateTime]::Now).days}} | where "Expires in (Days)" -lt 120 | fl} }

In another post I will expand on this topic and show how to generate CSRs, Import and Export Certificates, and renew certificates. I’m still doing research on those topics and will compile my information as soon as I can get it organized. Hope this helps!

IMPORTANT NOTE

POST COMMENTS! HAPPY TROUBLESHOOTING!

Download and install PowerShell PKI module from the PowerShell Gallery using PowerShell

Module Requirements

Cookies Notice for itblog.ldlnet.net

POST COMMENTS!
HAPPY TROUBLESHOOTING!