Wednesday, October 30, 2019

How to test SMTP using Opportunistic TLS with Powershell and grab the public certificate a SMTP server is using

Most email services these day employ Opportunistic TLS when trying to send Messages which means that wherever possible the Messages will be encrypted rather then the plain text legacy of SMTP.  This method was defined in RFC 3207 "SMTP Service Extension for Secure SMTP over Transport Layer Security" and  there's a quite a good explanation of Opportunistic TLS on Wikipedia https://en.wikipedia.org/wiki/Opportunistic_TLS .  This is used for both Server to Server (eg MTA to MTA) and Client to server (Eg a Message client like Outlook which acts as a MSA) the later being generally Authenticated.

Basically it allows you to have a normal plain text SMTP conversation that is then upgraded to TLS using the STARTTLS verb. Not all servers will support this verb so if its not supported then a message is just sent as Plain text. TLS relies on PKI certificates and the administrative issues that come around certificate management like expired certificates which is why I wrote this script. Essentially I wanted to see the Public Certificate that was in used by Recipient SMTP server and couldn't find any easy to use method to get it. Eg in a web browser you can always view a certificate to check its authenticity, but with SMTP there aren't a lot of good tools around for this, you can use Telnet to test in Plan text a SMTP server, but its not easy to retrieve the TLS public certificate from the server for inspection over Telnet (or using something like putty etc).

In PowerShell this is pretty easy back in 2006 I wrote a plain text SMTP test script https://gsexdev.blogspot.com/2006/09/doing-smtp-telnet-test-with-powershell.html and a variant to do alerting on verbs https://gsexdev.blogspot.com/2007/06/testing-smtp-verbs-and-sending-alert.html  so this is more just a modern version of this with the addition of using the System.Net.Security.SslStream class that supports creating the TLS connection and also allows you to easily export the recipient servers Public cert eg

        Write-Host("STARTTLS") -ForegroundColor Green
        $streamWriter.WriteLine("STARTTLS");
        $startTLSResponse = $streamReader.ReadLine();
        Write-Host($startTLSResponse)
        $ccCol = New-Object System.Security.Cryptography.X509Certificates.X509CertificateCollection
        $sslStream.AuthenticateAsClient($ServerName,$ccCol,[System.Security.Authentication.SslProtocols]::Tls12,$false);        
        $Cert = $sslStream.RemoteCertificate.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert);
        [System.IO.File]::WriteAllBytes($CertificateFilePath, $Cert);

I've created a small Powershell Script module that has two cmdlets the first is called Get-SMTPTLSCert which can be used to get the Public cert being used by the SMTP endpoint eg for Gmail you could use Get-SMTPTLSCert -ServerName smtp.gmail.com -Sendingdomain youdomain.com -CertificateFilePath c:\temp\gmailpubCer.cer By default this uses the client submission port 587 SMTP-MSA (Port 25 is often blocked from most locations) so its testing client(Message Submission Agent) to server (rather then server to server between to SMTP Mesage Transfer Agents). The Sending domain is required becuase most SMTP servers don't allows a empty helo/ehlo statement. I've also included a cmldet "Invoke-TestSMTPTLS" that does a test of the SMTP server and does Authentication if necessary. eg usually to use Port 587 you need to authenticate on the SMTP server. So in this instance once you have used the STARTTLS verb to upgrade the Plain text conversation to TLS you can then use the AUTH LOGIN verb to submit the username and password as base64 strings to authenticate eg this is what the Auth looks like in script
            $command = "AUTH LOGIN" 
            write-host -foregroundcolor DarkGreen $command
            $SSLstreamWriter.WriteLine($command) 
            $AuthLoginResponse = $SSLstreamReader.ReadLine()
            write-host ($AuthLoginResponse)
            $Bytes = [System.Text.Encoding]::ASCII.GetBytes($Credentials.UserName)
            $Base64UserName =  [Convert]::ToBase64String($Bytes)              
            $SSLstreamWriter.WriteLine($Base64UserName)
            $UserNameResponse = $SSLstreamReader.ReadLine()
            write-host ($UserNameResponse)
            $Bytes = [System.Text.Encoding]::ASCII.GetBytes($Credentials.GetNetworkCredential().password.ToString())
            $Base64Password = [Convert]::ToBase64String($Bytes)     
            $SSLstreamWriter.WriteLine($Base64Password)
            $PassWordResponse = $SSLstreamReader.ReadLine()
            write-host $PassWordResponse    

So the above code takes a PSCredential object and changes that into necessary SMTP verbs to authenticate.  So run against office365 this looks like


(The base64 values above decode to Useraname: and Password: )

This cmdlet doesn't actually send a Message it just invokes the envelope verbs and no Data verb is sent (where the MIME message would go). I've put a copy of the script on GitHub here https://github.com/gscales/Powershell-Scripts/blob/master/TLS-SMTPMod.ps1


Thursday, October 17, 2019

Doing Mailbox Change discovery with an EWS PowerShell Script

Mailbox Change discovery is the process of looking at any folders or items that are new or have been modified recently in a Mailbox. Its useful in a number of different ways including (but not limited to)

  • Looking at what objects a third party Addin is creating or modifying in your mailbox
  • Help to work out which FAI (Folder Associated Item) is being modified when changes are made to the configuration in Outlook or Outlook on the Web (this can be useful if you then want to automate those changes in your own scripts)
  • Fixing client issues caused by corrupt or bad items (eg if you've ever used MFCMapi to delete and Item that's causing a particular client function not to work correctly)
  • Getting an understanding of how the backend scaffolding of new features work in Outlook on the Web (eg looking at what the substrate Is doing in Office365) 
If you have ever looked recently at the Non_IPM Root folder of any Office365 Mailbox you can see by the large number of folders that are used by various different apps, substrate processes as well as for new client features there is a lot going on. So this script can also help give a bit of insight on what's happening in the background when you activate or use particular features (or potentially point you to the location in the Mailbox when your looking at problems that might be occurring with certain features)
I'll go through a specific use case later looking at the "contact favourite feature (which I struggled to even find the UI documentation for)" which is what prompted me to write this script.

What this script does

The script has three main functions

  1. Enumerates every folder in the Mailbox (both IPM and NON_IPM_Subtree) as well as Search Folders and looks at the created and modified date of each folder. If they where created or modified in the lookbacktime then it's adds them to the report
  2. It then does a Scan of the Items in each Folder (excluding Search Folders) and if it finds any items that where modified or created after the lookbacktime then it adds these to the report
  3. It then does a Scan of the FAI Items (Folder Associated Items) in the Folder and again if the items where modified or created after the lookbacktime then it adds these to the report
The Output of the Report then contains information about what folders, Items and FAI Items have either been created or modified in the Mailbox in the last x number of seconds.

An Example

The best way to demonstrate this is with an Example which was the reason I wrote the script, The Contact Favourite feature in Outlook on the Web gives you the ability to click the star next to a Contacts name in OWA which then creates a favourite Shortcut eg


So I wanted to know when you did this where does the favourite item get created, what information it was storing and what other changes where happening. So this is where the following script comes into handy to find this information out all I needed to do was favourite a contact and then run the script immediately after to look at the items which changed in the Mailbox in the last 60 seconds. Eg a run of the script after I made the above change yielded a report that looked like



So from the above report you can see that firstly a new Search Folder was created 
\FavoritePersonas\Glen Scales_7d09f835-0028-4bd7-bed9-59535127bbe1 New SearchFolder



under the \FavoritePersonas\ directory and also an object of type SDS.32d4b5e5-7d33-4e7f-b073-f8cffbbb47a1.OutlookFavoriteItem was created under the folder

\ApplicationDataRoot\32d4b5e5-7d33-4e7f-b073-f8cffbbb47a1\outlookfavorites

The other thing that I included in the report was the EntryId of the Item found so in the above case i can take the EntryId for the outlookfavourite and open the Item in a Mapi editor like OutlookSpy or MFCMAPI eg


And you can then see all the MAPI properties on the Item (or delete/export etc)



That's it relatively simple to use eg

Invoke-MailboxChangeDiscovery -MailboxName mailbox@domain -secondstolookback 60

I've put this script up on GitHub here https://github.com/gscales/Powershell-Scripts/blob/master/ChangeDiscovery.ps1

Friday, October 04, 2019

Using the MSAL (Microsoft Authentication Library) in EWS with Office365

Last July Microsoft announced here they would be disabling basic authentication in EWS on October 13 2020 which is now a little over a year away. Given the amount of time that has passed since the announcement any line of business applications or third party applications that you use that had been using Basic authentication should have been modified or upgraded to support using oAuth. If this isn't the case the time to take action is now.

When you need to migrate a .NET app or script you have using EWS and basic Authentication you have two Authentication libraries you can choose from

  1. ADAL - Azure AD Authentication Library (uses the v1 Azure AD Endpoint)
  2. MSAL - Microsoft Authentication Library (uses the v2 Microsoft Identity Platform Endpoint)
the most common library you will come across in use is the ADAL libraries because its been around the longest, has good support across a number of languages and allows complex authentications scenarios with support for SAML etc. The MSAL is the latest and greatest in terms of its support for oAuth2 standards and is where Microsoft are investing their future development efforts. A good primer for understanding the difference in terms of the Tokens that both of these endpoint generate is to read https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens

So which should you choose ? If your using PowerShell then the ADAL is the easiest to use and there are a lot of good examples for this like. However from a long term point of view using MSAL library can be a better choice as its going to offer more supportability (new features etc) going forward as long as you don't fall into one of  the restrictions described in https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/Adal-to-Msal

In this post I'm going to look at using the MSAL library with EWS to access Mailboxes in Exchange Online. 

Scopes

One of the biggest differences when it comes to coding between the libraries with ADAL you specify the resource your going to use eg "https://outlook.office365.com" and with the MASL you specific the scopes you are going to use. With EWS its relatively simple in that  there are only two scopes (EWS doesn't allow you to constrain your access to different mailbox item types) which you would first need to allow in your Application registration which can be found in the Supported Legacy API's section of the application registration(make sure you scroll right to the bottom)

Delegated Permissions


Application Permissions (where you going to use AppOnly Tokens)

.default Scope

For v.1 apps you can get all the static scopes configured in an application using the .default scope so for ews that would look something like https://outlook.office365.com/.default . When your using App Only tokens this becomes important.


App registration and Consent

One of the advantages of the MSAL library is dynamic consent which for EWS because in practice your only going to be using one scope it doesn't have much use. However if your also going to be using other workloads you maybe able to take advantage of that feature. For the app registration you need to use the v2 Endpoint registration process (which is the default now in the Azure portal) see https://docs.microsoft.com/en-us/graph/auth-register-app-v2. This also makes it easy to handle the consent within a tenant.

Getting down to coding

In the ADAL there was only one single class called the AuthenticationContext which you used to request tokens. In the MSAL you have the PublicClientApplication (which you use for standard user authentication) and ConfidentialClientApp which gets used for AppOnly tokens and On-Behalf-Of flow.

Endpoints 

With the v2 Endpoint you have the option of allowing
  1. common
  2. organizations
  3. consumers
  4. Tenant specific (Guid or Name)
For EWS you generally always want to use the Tenant specific endpoint which means its best to either dynamically get the TenantId for the tenant your targeting or hard code it . eg you can get the TenantId need with 3 lines of C#


string domainName = "datarumble.com";
HttpClient Client = new HttpClient();
var TenantId = ((dynamic)JsonConvert.DeserializeObject(Client.GetAsync("https://login.microsoftonline.com/" + domainName + "/v2.0/.well-known/openid-configuration")
     .Result.Content.ReadAsStringAsync().Result))
    .authorization_endpoint.ToString().Split('/')[3];
In PowerShell you can do it with

$TenantId = (Invoke-WebRequest https://login.windows.net/datarumble.com/v2.0/.well-known/openid-configuration | ConvertFrom-Json).token_endpoint.Split('/')[3]

Delegate Authentication in EWS with MSAL and the EWS Managed API

This generally is the most common way of using EWS where your authenticating as a standard User and then accessing a Mailbox. If its a shared Mailbox then access will need to be have granted via Add-MailboxFolderPermission or you using EWS Impersonation

This is the simplest C# example of an Auth using the MSAL library in a Console app to logon (the currently logged on user).

 string MailboxName = "gscales@datarumble.com";
 string scope = "https://outlook.office.com/EWS.AccessAsUser.All";
 string redirectUri = "msal9d5d77a6-fe09-473e-8931-958f15f1a96b://auth";
 string domainName = "datarumble.com";

 HttpClient Client = new HttpClient();
 var TenantId = ((dynamic)JsonConvert.DeserializeObject(Client.GetAsync("https://login.microsoftonline.com/" + domainName + "/v2.0/.well-known/openid-configuration")
  .Result.Content.ReadAsStringAsync().Result))
  .authorization_endpoint.ToString().Split('/')[3];

 PublicClientApplicationBuilder pcaConfig = PublicClientApplicationBuilder.Create("9d5d77a6-fe09-473e-8931-958f15f1a96b")      
  .WithTenantId(TenantId);
   
 pcaConfig.WithRedirectUri(redirectUri);
 var TokenResult = pcaConfig.Build().AcquireTokenInteractive(new[] { scope })
  .WithPrompt(Prompt.Never)
  .WithLoginHint(MailboxName).ExecuteAsync().Result;

 ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2016);
 service.Url = new Uri("https://outlook.office365.com/ews/exchange.asmx");
 service.Credentials = new OAuthCredentials(TokenResult.AccessToken);
 service.HttpHeaders.Add("X-AnchorMailbox", MailboxName);

 Folder Inbox = Folder.Bind(service, WellKnownFolderName.Inbox);


AppOnly Tokens

This is where your Application is authenticating using a App Secret or SSL certificate, after this your App will get full access to all Mailboxes in a tenant (it important to not that the scoping feature https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access doesn't work with the EWS so you need to be using the Graph or Outlook api).

 string clientId = "9d5d77a6-fe09-473e-8931-958f15f1a96b";
 string clientSecret = "xxxx";
 string mailboxName = "gscales@datarumble.com";
 string redirectUri = "msal9d5d77a6-fe09-473e-8931-958f15f1a96b://auth";
 string domainName = "datarumble.com";
 string scope = "https://outlook.office365.com/.default";

 HttpClient Client = new HttpClient();
 var TenantId = ((dynamic)JsonConvert.DeserializeObject(Client.GetAsync("https://login.microsoftonline.com/" + domainName + "/v2.0/.well-known/openid-configuration")
  .Result.Content.ReadAsStringAsync().Result))
  .authorization_endpoint.ToString().Split('/')[3];

 IConfidentialClientApplication app = ConfidentialClientApplicationBuilder.Create(clientId)
  .WithClientSecret(clientSecret)
  .WithTenantId(TenantId)
  .WithRedirectUri(redirectUri)
  .Build();

  
 var TokenResult = app.AcquireTokenForClient(new[] { scope }).ExecuteAsync().Result;
 ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2016);
 service.Url = new Uri("https://outlook.office365.com/ews/exchange.asmx");
 service.Credentials = new OAuthCredentials(TokenResult.AccessToken);
 service.HttpHeaders.Add("X-AnchorMailbox", mailboxName);
 service.ImpersonatedUserId = new ImpersonatedUserId(ConnectingIdType.SmtpAddress, mailboxName);
 Folder Inbox = Folder.Bind(service, new FolderId(WellKnownFolderName.Inbox, mailboxName));


Token Refresh

One of the big things missing in the EWS Managed API is a callback before each request that checks for an expired Access Token. Because tokens are only valid for 1 hour if you have a long running process like a migration/export or data analysis then you need to make sure that you have some provision in your code to track the expiry of the access token and the refresh the token when needed.

Doing this in PowerShell

If your using PowerShell you can use the same code as above as long as import the MSAL library dll into your session

Some simple auth examples for this would be

 Delegate Authentication 


$MailboxName = "gscales@datarumble.com";
$ClientId = "9d5d77a6-fe09-473e-8931-958f15f1a96b"
$scope = "https://outlook.office.com/EWS.AccessAsUser.All";
$redirectUri = "msal9d5d77a6-fe09-473e-8931-958f15f1a96b://auth";
$domainName = "datarumble.com";
$Scopes = New-Object System.Collections.Generic.List[string]
$Scopes.Add($Scope)
$TenantId = (Invoke-WebRequest https://login.windows.net/datarumble.com/v2.0/.well-known/openid-configuration | ConvertFrom-Json).token_endpoint.Split('/')[3]
$pcaConfig = [Microsoft.Identity.Client.PublicClientApplicationBuilder]::Create($ClientId).WithTenantId($TenantId).WithRedirectUri($redirectUri)
$TokenResult = $pcaConfig.Build().AcquireTokenInteractive($Scopes).WithPrompt([Microsoft.Identity.Client.Prompt]::Never).WithLoginHint($MailboxName).ExecuteAsync().Result;

AppOnly Token


$ClientId = "9d5d77a6-fe09-473e-8931-958f15f1a96b"
$MailboxName = "gscales@datarumble.com"
$RedirectUri = "msal9d5d77a6-fe09-473e-8931-958f15f1a96b://auth"
$ClientSecret = "xxx";
$Scope = "https://outlook.office365.com/.default"
$TenantId = (Invoke-WebRequest https://login.windows.net/datarumble.com/v2.0/.well-known/openid-configuration | ConvertFrom-Json).token_endpoint.Split('/')[3]
$app =  [Microsoft.Identity.Client.ConfidentialClientApplicationBuilder]::Create($ClientId).WithClientSecret($ClientSecret).WithTenantId($TenantId).WithRedirectUri($RedirectUri).Build()
$Scopes = New-Object System.Collections.Generic.List[string]
$Scopes.Add($Scope)
$TokenResult = $app.AcquireTokenForClient($Scopes).ExecuteAsync().Result;