Monday, December 17, 2018

ZAP (Zero-hour auto purge) Junk email reporting for Office365 using EWS and REST

Zero-hour auto purge is one of the features of Office365 that will detect malicious and Spam emails and move them to the Junk email folder for any email that has breached the first level defences and has been delivered to users mailboxes. There is a good description of how it works here but basically when the service learns a particular message was malicious/spam it can retrospectively detect and eliminate/move any simular messages that arrived previously and weren't detected.

This is a good and much need feature as no AntiSpam or Malware solution is perfect (no matter what the vendor say) so there will always be the case where thing slip through. But this very fact is what causes an exposure point where the potentially malicious email sits in the Inbox of end user up until the time its gets zapped. What I wanted to present in this post is a few ways you can measure the amount of the time you may have been vulnerable for and show some methods you can use to look more at messages and it's potential malicious content.

How to detect messages that have been zapped in a EWS and REST script

There is a good reference article for this here , what happens when a Message is Zapped and moved to the Junk Email folder is a Internet Message Headers is added which will also create a underlying Extended property see



We can use  this in a EWS or REST script to do some reporting on. In EWS we can use an Exists Search filter on Messages in the JunkEmail folder to find just messages where this property has been set meaning that these messages have been zapped

 $ZapInfo = new-object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition([Microsoft.Exchange.WebServices.Data.DefaultExtendedPropertySet]::Common, "X-Microsoft-Antispam-ZAP-Message-Info", [Microsoft.Exchange.WebServices.Data.MapiPropertyType]::String)
 $Sfexists = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+Exists($ZapInfo) 

In the Graph API we can also do something simular using the following filter

$filter=singleValueExtendedProperties/any(ep: ep/id eq 'String {00062008-0000-0000-c000-000000000046} Name X-Microsoft-Antispam-ZAP-Message-Info' and ep/value ne null) 

What this does is returns us a collection of Messages that have been zapped, I went a step further in my script and put a DateTime filter around this as well (but technically the Junk Email folder has a default retention period of 30 days so it shouldn't really have a large volume of email). Once you have messages return if you check the datetime the message was received and then using the Time-Span function in PowerShell calculate the TimeSpan  against last modified time (which should have been the time the Message was Zapped and moved to the JunkEmail folder) this will give you a good indication of the time that these messages sat in the Inbox of the user (this becomes your vulnerability period). You can also look at the read setting of the Email to determine if the user had actually read the Message that was Zapped. I've gone a another step further in my reporting script to also do some Antispam analysis of the email headers using some code I previously wrote so you can also look the DKIM,DMARC etc values this mail received as well. So in the end what my reporting script does is checks for Zapped messages in the Junk Email folder for a specific time period and then produces a report on the actually exposure time and relevant information around this so it can be further evaluated. The output of the report is something like this

I've put the EWS script that can do this report on GitHub https://github.com/gscales/Powershell-Scripts/blob/master/ZapStatistics.ps1 To run a Report on a particular mailbox use

 Get-ZapStatistics  -MailboxName mailbox@datarumble.com -startdatetime (Get-Date).AddDays(-14)  | Export-Csv -Path c:\Reports\mailboxName.csv -NoTypeInformation

I've also added the same type of script to my Exch-Rest Module so you can do the same thing using the Microsoft Graph API

 Get-EXRZapStatistics  -MailboxName mailbox@datarumble.com -startdatetime (Get-Date).AddDays(-14)  | Export-Csv -Path c:\Reports\mailboxName.csv -NoTypeInformation

The module is available from the PowerShell Gallery or GitHub

DevOps - Looking at deeper analysis and proactive measure

I thought I'd start including a DevOps section in some of my posts to show how you can delve a bit deeper to look what we are reporting on and some potential proactive measures you might be able to put in place to earn such a DevOps tag. In this section I'm assuming you know enough about development to know your way around objects,methods,properties etc. I'm going to be using my Exch-Rest module because its the best tool I have to do this and its also a good way to improve the module itself (and its free).

Analysis 

So first lets just look at one of the Messages that have been zapped to do this just pull the Messages into a collection like

$Messages =  Get-EXRZapStatistics  -MailboxName mailbox@datarumble.com -startdatetime (Get-Date).AddDays(-14)

Then dump out the first message in the Collection

$Messages[0]

This give us something like



So we have a whole bunch of interesting information about the source that is trying to essentially attack or steal the users credentials. We have

Source SMTP server IP Address in the CIP and SPF Values
We have the Reverse DNS PTR (which tells us the sending server country location)
The HostName of the SMTP server which does resolve in DNS to the same IP Address as the PTR record but the HostName doesn't match
The SCL value of 1 which is pretty low

I can see the message was Read by the user (which isn't good if this is a user). So the next thing that might be useful is looking at the content of the messages to see if it has any attachments. We can do this using the InternetMessageId and the following

$message = Find-EXRMessageFromMessageId -MailboxName gscales@datarumble.com -MessageId "<57a196603977b4b3d9e0680d3119d816 baseny-wroclaw.pl="">"

This gives us some information like the following


So this tells us right away there where no Attachments so that is one less thing to worry about it also shows the they tried to fool the target user by putting in a different email address in the SenderName then the actual senders Email address as they where trying to make it appear as if it was a system generated message that came from Microsoft itself.

So now we know that there where no attachments it probably just has a link they want the user to click. To get more information on links in the body of the Message we can use something like the following

$expandedEmail = Get-EXREmailBodyLinks -MailboxName gscales@datarumble.com  -InternetMessageId "<57a196603977b4b3d9e0680d3119d816 baseny-wroclaw.pl="">"

and then we can view the links in the email like


Bingo there you have an attempt to fake the domain name to make it look like something that a user may have used before so they might feel confident entering their details into etc. At this point you might want to trawl through what ever logs you have available to see if a user did actually visit that URL and take some urgent action if they did.

Being Proactive

So what I've gone through above are some manual steps above to show that the while you can be assured that the cloud is doing its job to a certain extent there are some extra measures you can take to keep a closer a eye on what potential these malicious email are trying to do and catch any unsuspecting users before something like this becomes a bigger problem for you. The cloud isn't set and forget and if your not doing some extra checks on what is happening in the service your not being as an effective for the company you work for (or on behalf of) as you can be. Some other quick DevOps ideas

  • Setup a Azure Runbook that uses Certificate Authentication and AppOnly tokens that will report on the Zapmessages and do some further automated analysis on a Daily or Weekly basis.
  • Look for correlations based on the information you have in the Zapped messages, if your being targeted by someone there is a good chance that location data, Ipaddress from one attacked maybe used in subsequent ones. So apply those patterns to look at your Message Tracking logs which will give you details about SenderIp ect.
Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365,Microsoft Teams, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com (nothing too big or small).   




Monday, December 10, 2018

Updates to the Exch-Rest PowerShell Module to support PowerShell Core, Azure Cloud Shell and more ADAL integration options

I've had some time recently to do some much needed updates to my Exch-Rest module so it now supports both Azure Cloud Shell and PowerShell Core on Linux (tested on RHEL,CentOS, Debian and Ubuntu). So now you can logon to an Office365 Mailbox using this Module with Powershell on Linux and send Email or a Skype for Business Message or do some mailbox reporting eg


The requirements on Linux is you need to be using the latest version of PowerShell core installed as per https://docs.microsoft.com/en-us/powershell/scripting/setup/installing-powershell-core-on-linux?view=powershell-6 .This ensures that all the required .net Core libraries will be available as older version of .Net core didn't have some of the libraries I'm using and I didn't want to backport for older versions.  Also because there are no Linux forms to interact with for authentication you need to pass in the credentials to use via a PSCredential and the code will use the password grant to get the Token eg
$cred = Get-Credential -UserName gscales@datarumble.com
connect-exrmailbox -MailboxName gscales@datarumble.com -Credential $Cred
Azure Cloud Shell

As Cloud Shell is a browser based version of  PowerShell core running on Linux the same connection method of using the credentials as above is needed.

ADAL Integration

I've also added full integration with the ADAL library for authentication so in addition to the native dependency free script methods the module now distributes the ADAL libraries and supports Authentication using that library as well as Token refreshes for scripts that run over an hour etc (using Acquiretokenasync in the ADAL). This supports the following scenarios such as
To use the ADAL libraries for Logon use the following 

connect-exrmailbox -MailboxName gscales@datarumble.com -useADAL

To use the Never Prompt to use the ADAL Cache
connect-exrmailbox -MailboxName gscales@datarumble.com -useADAL -Prompt Never
For connecting using the currently logged on credentials use
connect-exrmailbox -MailboxName gscales@datarumble.com -useADAL -useLoggedOnCredentials -AADUserName gscales@datarumble.com

(The -AADUserName variable is optional but usually required read the GitHub link in the second bullet point)

For those who want to do something simular and are using EWS you will need something like the following to get the AccessToken using ADAL in a normal PowerShell Script.

    $ClientId = "d3590ed6-52b3-4102-aeff-aad2292ab01c"
    $ResourceURI = "https://outlook.office365.com"
    $EndpointUri = 'https://login.microsoftonline.com/common'
    $Context = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext($EndpointUri)
    $AADCredential = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.UserCredential" -ArgumentList "username@domain.com"
    $authResult = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContextIntegratedAuthExtensions]::AcquireTokenAsync($Context, $ResourceURI, $ClientId, $AADcredential)
    if ($authResult.Result.AccessToken) {
        $token = $authResult.Result
    }
    elseif ($authResult.Exception) {    
        throw "An error occured getting access token: $($authResult.Exception.InnerException)"    
    }
    return $token

This also goes along with support for using the Module in an Azure Runbook which was added last month.

The Exch-REST Module is available from the PowerShell Gallery https://www.powershellgallery.com/packages/Exch-Rest and GitHub https://github.com/gscales/Exch-Rest

A big thankyou to all those people who provided feedback on using the module hopefully these upgrades make it easier to use in more scenarios.

Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365,Microsoft Teams, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com (nothing too big or small).

Wednesday, November 21, 2018

Scripters guide to using Guest Access in Office365 to automate things

Guest access is one of the ways in Office365 of collaborating between different organizations which allows you to give certain people who are outside of your company access to a limited subset of the resources you have in the Cloud. This can be an Office365 unified Group or Microsoft Team but also other workloads like SharePoint and OneDrive can utilize this.
When it comes to scripting there are a number of value add things you can do to automate tasks for different people who have guest accounts in another tenant. The first step to automating with Guest Access is to Authenticate and generate an access token in the Guest tenant.
Getting the Guest Tenants Authorization Endpoint
Before you can authenticate you need to first obtain the Guest tenants Authorization endpoint for the tenant where the Guest Account exists in. To do this you can make a simple Get Request like the following
Invoke-WebRequest -uri ("[https://login.windows.net/{0}/.well-known/openid-configuration" -f "guestdomain.com")
this will return a JSON result that contains the Authorization endpoint for the guest tenant along with other information useful when authenticating.
While Invoke Web Request will do the job fine if you ever what to execute something like this from an Azure Run-book it better to use the httpclient object instead. Here is a simple function to get the Authorization endpoint
function Get-TenantId {
param( 
    [Parameter(Position = 1, Mandatory = $false)]
    [String]$DomainName
   
)  
Begin {
    $RequestURL = "https://login.windows.net/{0}/.well-known/openid-configuration" -f $DomainName
    Add-Type -AssemblyName System.Net.Http
    $handler = New-Object  System.Net.Http.HttpClientHandler
    $handler.CookieContainer = New-Object System.Net.CookieContainer
    $handler.AllowAutoRedirect = $true;
    $HttpClient = New-Object System.Net.Http.HttpClient($handler);
    $Header = New-Object System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json")
    $HttpClient.DefaultRequestHeaders.Accept.Add($Header);
    $HttpClient.Timeout = New-Object System.TimeSpan(0, 0, 90);
    $HttpClient.DefaultRequestHeaders.TransferEncodingChunked = $false
    $Header = New-Object System.Net.Http.Headers.ProductInfoHeaderValue("Get-TenantId", "1.1")
    $HttpClient.DefaultRequestHeaders.UserAgent.Add($Header);
    $ClientResult = $HttpClient.GetAsync([Uri]$RequestURL)
    $JsonResponse = ConvertFrom-Json  $ClientResult.Result.Content.ReadAsStringAsync().Result
    return $JsonResponse.authorization_endpoint
}}
Using the ADAL
Once you have the authorization endpoint your ready to Authenticate, using the ADAL library which is a popular method you would use something like the following (where I’m using the above function to get the endpoint in @line3
Import-Module .\Microsoft.IdentityModel.Clients.ActiveDirectory.dll -Force
$PromptBehavior  =  New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.PlatformParameters -ArgumentList Auto
$EndpointUri  =  Get-Tenantid -DomainName domain.com
$Context  =  New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext($EndpointUri)
$token  = ($Context.AcquireTokenAsync("https://graph.microsoft.com","d3590ed6-52b3-4102-aeff-aad2292ab01c","urn:ietf:wg:oauth:2.0:oob",$PromptBehavior)).Result
In the above example I’ve used the preapproved Office appId otherwise if you where to use your own AppId that would need to be authorized in the Guest tenant (which is a always a degree of difficulty when dealing with another companie's IT departments).
Once you have the Token you can then make a Request to the ./me endpoint to find a bit more about your account in the guest tenant eg
$Header = @{
        'Content-Type'  = 'application\json'
        'Authorization' = $token.CreateAuthorizationHeader()
        }
Invoke-RestMethod -Headers $Header -Uri "https://graph.microsoft.com/v1.0/me" -Method Get -ContentType "application/json" 
Or to get the Groups or Teams your a member of you can use
Invoke-RestMethod -Headers $Header -Uri "https://graph.microsoft.com/v1.0/me/memberOf" -Method Get -ContentType "application/json" 
And to get the Members or a particular Team or Group
Invoke-RestMethod -Headers $Header -Uri "https://graph.microsoft.com/v1.0/groups/938383f7-3060-4604-b3a5-cbdb0a5fc90f/members" -Method Get -ContentType "application/json"
Where the Guid (938383f7-3060-4604-b3a5-cbdb0a5fc90f in this instance) is the id you retrieved when you used me/memberOf
This gives you access to all the raw data about each of the members of a Group you might be interacting with. For some real life uses of this take a look at the module section below
Other things you can do with this which I’ll go through below are
  • Export the Group members from  a Guest Teams/Group to CSV
  • Download or Upload Files to shared Team/Group drive
  • Export the Groups Calendar to a CSV file

Using a Module

If your looking for an easier way of using Guest Access check out my Exch-Rest Module on the PowerShell Gallery https://www.powershellgallery.com/packages/Exch-Rest/3.22.0 and gitHub https://github.com/gscales/Exch-Rest . The following are samples for this module
To connect to a tenant as a Guest use
 Connect-EXRMailbox -GuestDomain guestdomain.com -MailboxName gscales@datarumble.com
You can then execute the Me and MemberOf requests using
Get-EXRMe

Get-EXRMemberOf
Export the members of a Group or Team your a member of as a Guest to CSV
The following can be used to Export the members of a Team or Unified Group in a Guest tenant to a CSV file. The Inputs you need a the SMTP address of the Group which you can get from running Get-EXRMemberOf in the Guest Tenant
$Group = Get-EXRUnifedGroups -mail guest@guestdomain.org
$GroupMembers = Get-EXRGroupMembers -GroupId $Group.id -ContactPropsOnly
$GroupMembers |  Export-Csv -path c:\temp\groupexport.csv -noTypeInformation
If you wish to include the user photo in the export you can use (although the AppId you use to connect must have access to the userphoto)
$Group = Get-EXRUnifedGroups -mail guest@guestdomain.org
$GroupMembers = Get-EXRGroupMembers -GroupId $Group.id -ContactPropsOnly
$GroupMembers |  Export-Csv -path c:\temp\groupexport.csv -noTypeInformation
Downloading a File from Group/Teams Shared Drive (Files) as a Guest
$Group = Get-EXRUnifedGroups -mail guest@guestdomain.org
$FileName = "thenameofthedoc.docs"
$File = Get-EXRGroupFiles -GroupId $Group.id -FileName $FileName
Invoke-WebRequest -Uri $File.'@microsoft.graph.downloadUrl' -OutFile ("C:\downloaddirectory\$FileName)
Export a Groups Calendar to CSV as a Guest
$Group = Get-EXRUnifedGroups -mail guest@guestdomain.org
Get-EXRGroupCalendar -GroupId $Group.id -Export | export-csv -NoTypeInformation -Path c:\temp\calendarexport.csv
By default the next 7 days is exported by the time windows can be tweaked using -starttime and -sndtime parameter in the Get-EXRGroupCalendar cmdlet
Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365,Microsoft Teams, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com (nothing too big or small).

Tuesday, November 13, 2018

Thursday, November 01, 2018

Reporting on Teams private Chat Activity using the TeamChat Folder in Exchange

This is a Segway from my last post on Reporting on Skype for Business Messaging Activity using the Conversation History Folder in Exchange .In this Post I'm going to be looking at the Private chat messages from Microsoft Teams that get stored in a hidden folder called TeamChat (see this post for how to get that folder in EWS) under the conversation History Folder as part of the compliance process for Teams.  As Teams is still a work in process a lot of those compliance properties have changed since my first post about getting this folder and looking at the history of even the limited dataset in my mailboxes there is a lot of fluidity in the compliance information that is being added (so its a little bit of a dogs breakfast in terms of data when you go back a number of months). The compliance properties themselves get added when the substrate process in the cloud copies the Chat Message from the skypespaces backend to the Mailbox.  To give you a visual representation on what these properties look like you can use a MAPI Editor 



Unlike Skype where there was one message that would represent multiple messages in a Chat Thread, with Teams there is a one to one basis for each private Chat message in a Thread. All the interesting meta-information that is needed for reporting is held in the above Extended properties. Take for instance the ToSkypeInternalIds property which contains the list of the recipients for Private Chats stored in a Multivalued string property. (this information also gets stored in the Recipient Collection of the Message in a resolved format)


Instead of SIP addresses like Skype, (for chat anyway) Teams has its own format which contains the AzureAd guid of the user (or Guest). If you want  to resolve this back into an EmailAddress or UPN the Graph API can be used to do this eg if I take a Teams address

8:orgid:fb25746f-fa0c-4f01-b815-a7f7b878786f

using a simple Split to get just the guid portion of this string you can then do a simple Rest Get to get the user information as long as you have an underlying Token to use in the Graph API  eg

$GUID = "8:orgid:fb25746f-fa0c-4f01-b815-a7f7b878786f".Split(':')[2] "https://graph.microsoft.com/v1.0/users('" + $GUID + "')"

I've put together a script that first gets the TeamChat folder using EWS then enumerates the messages in that folder for a particular day time frame and using a number of the extended properties from the first screen shot to build some log entries that can then be summarised into some reports. 

EWS Meet Graph

Because EWS doesn't have a way of resolving the Guids from the From and To Address properties in Teams. I've integrated a bit of Microsoft Graph Code to the do the resolution. Because by default this script uses oAuth we can take the Refresh Token from the first auth used to get into EWS and then get another token for the GraphEndpoint without needing to re-prompt for authentication so this all fits together nicely.


So by default when you run the script you will get flat log entries back like


Or you can Group the threads together (based on the ThreadId) using the -GroupThreads Switch


Like the Conversation History script I have 3 base reports that look something like this in HTML



To use these Reporting options

Get-TeamsIMLog -MailboxName gscales@datarumble.com -Days 30 -reportbyDate 
-htmlReport | Out-file c:\temp\30dayreport.html
if you want the same report in csv use

Get-TeamsIMLog -MailboxName gscales@datarumble.com -Days 30
 -reportbyDate | ConvertTo-Csv -NoTypeInformation -path c:\temp\30dayreport.csv 
To report the top senders that sent this Mailbox in Direct Chats

Get-TeamsIMLog -MailboxName gscales@datarumble.com -Days 30
 -reportFrom
To report the top recipients of IM's


Get-TeamsIMLog -MailboxName gscales@datarumble.com -Days 30
 -reportTo
I've put the code for this script up on Git Hub here 

Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365,Microsoft Teams, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com(nothing too big or small).