Thursday, April 26, 2018

Looking at the Email sentiment analysis properties in Office365 with EWS and REST

Sentiment analysis which means looking at a block of text and telling if this is Negative or Positive (okay oversimplified but the Wikipedia page has a good basic introduction into this https://en.wikipedia.org/wiki/Sentiment_analysis) has been one of the standout applications of Machine learning for the last couple of years. If your a observer of what MAPI properties are available on Messages in your Inbox in Office365 like me you may have noticed a new one appear recently on some messages which is the Sentiment analysis of the body Text of an email eg


Before I go any further everything I'm talking about in this post is from observation only and none of this as far as I know is documented anywhere. The entity extraction part of this is documented and is a relative new feature that can be used in Addin's (or other applications) https://docs.microsoft.com/en-us/outlook/add-ins/match-strings-in-an-item-as-well-known-entities . Entity extraction is one of those cool power up features that allows you to write more useful Addins with a smaller amount of code and coding effort(eg if you had to parse this stuff out yourself and do the analysis it would be a lot of time/effort/testing etc)  so this type of machine learning extraction is just another evolution of this.

What you find in this named property (when it is set which is not always the case) is a JSON payload with the emotiveprofile returned from the ML process. If you going to use the in EWS or REST because of the size of the payload you will need to do a GetItem (in EWS) or a single Item GET in REST. For performance reasons when you do this with either of these API's you should take advantage of the Batching feature available. Batching is a little new in the Graph API and allows you to retrieve up to 20 items in a single call, in EWS there is no limit but realistically you should keep you calls to under 50 to ensure you don't get throttled see .

I've put together two samples of using this property the first is issuing EWS, for this I've modified the Generic Folder Item enum script I recently posted. To returned the sentiment property using this script just use the -ReturnSentiment switch. For REST I've modified my Exch-Rest library to be able to return this property which included implementing batching in the library to give the best performance.  What both of these scripts do is request the Extended property relating to the sentiment analysis and then parse the JSON using the ConvertFrom-Json cmdlet. The data is then published as first class properties on the Item with the polarity (eg Positive or Negative) published as a separate property.


Some examples of these in action

EWS - Source Script https://github.com/gscales/Powershell-Scripts/blob/master/GenericFolderAndItem.ps1

Invoke-GenericFolderItemEnum -MailboxName gscales@datarumble.com -FolderPath \Inbox -Credentials $creds -MaxCount 5 -ReturnSentiment -FullDetails | Where-Object {@_.Sentiment -ne $null} |  Select Subject,Sentiment,EmotiveProfile | fl


REST - Source Exch-REST Module https://www.powershellgallery.com/packages/Exch-Rest and https://github.com/gscales/Exch-Rest
https://github.com/gscales/Exch-Rest
The -ReturnSentiment switch in Get-EXRLastInboxEmail, Get-EXRWellknowFolderItems and Get-EXRFolderItems will return the extracted sentiment and EmotiveProfile from the messages


Get-EXRLastInboxEmail -MailboxName jcool@datarumble.com -Focused -ReturnSentiment | Select Subject,Sentiment,EmotiveProfile | fl





Tuesday, April 17, 2018

Generic Toolbox script for using EWS to enumerate Folders and Items in an Exchange Mailbox

There's a old saying that a tradesmen is only as good as their tools and the same can be said for ITPro's and developers. You can save yourself a lot of time when investigating particular tasks you want a script by having something ready built that can do 90% of task at hand.  I've decided to share a script that I use quite often during the investigation phase of any EWS scripting tasks that allows you to generically grab a folder (or collection of Folders) and the Items within those folder. While this is something that can be put together quite quickly together using snippets there always tends to be extra properties that you need for different tasks so the generic script I'm showing today includes all the niceties kind of like having the right sized spanner rather the shifter.

So the script in question can be found on GitHub here https://github.com/gscales/Powershell-Scripts/blob/master/GenericFolderAndItem.ps1

So what can it do

Connect and show the properties on a Mailbox Folder eg
Invoke-GenericFolderConnect -MailboxName mailbox@domain.com -FolderPath \Inbox -Credentials $cred


So in the above example this is my Inbox, some of the extra things I've added in this script over what you would normally see in EWS is to promote the ExtendedProperties for the retentionTags,FolderSize,FolderPath and AttachmentCount to first class folder properties on the object so they are much easy to first view but to also script on top off. Eg say I only want to look at the Inbox and Subfolders of the Inbox there where larger then 10 MB I can do the following
Invoke-GenericFolderConnect -MailboxName gscales@datarumble.com -FolderPath \Inbox -Credentials $cred  -Recurse | Where-Object{$_.FolderSize -gt 10mb} | select FolderPath,FolderSize
The above examples uses the -Recurse switch parameter of the script to recurse Subfolders of the pass in folder Path you connected to and then makes use of those promoted properties using the pipeline and Where-Object in PowerShell.

One other thing I've added to the Standard EWS FolderObject is the GetLastItem ScriptMethod which does what is says by getting the last item in a folder. For example if I wanted to get the Last Item in My Inbox I could use the following


I could use this method in a recursion as well to show me all the folders in my mailbox that have received a Mail in the last week eg

Invoke-GenericFolderConnect -MailboxName gscales@datarumble.com -RootFolder -Credentials $cred -Recurse | Where-Object{$_.GetLastItem().DateTimeReceived -gt (Get-Date).AddDays(-7)} | Select-Object FolderPath,FolderSize,@{Name="DateTimeReceived"; Expression = {$_.L
astItem.DateTimeReceived}},@{Name="Subject"; Expression = {$_.LastItem.Subject}})}



Enumerating FolderItems

As I've show above getting the Last Item in a folder can be useful but a lot of the time you will want to enumerate all the Items with a folder (or a least the Top X) to do particular reporting or investigation tasks. To do you use the following generic function which will give you back all the items in the Inbox.


Invoke-GenericFolderItemEnum -MailboxName gscales@datarumble.com -FolderPath \Inbox -Credentials $creds

To  limit the results to just the top 10 items in the Folder use the -MaxCount switch eg

Invoke-GenericFolderItemEnum -MailboxName gscales@datarumble.com -FolderPath \Inbox -Credentials $creds -MaxCount 10
EWS by default only returns certain properties while doing an item enumeration (using findItems) so things like the Recipients and the Message body wont be returned by default for performance reasons. If you wanted to view these properties on a Message you could use the Load() method on the Message itself or you can use the -FullDetails Switch which will batch these operations which will make it perform better if you have a large number of items that you wish to do this on. Eg to show the last 10 messages in the Inbox with FullDetails


Invoke-GenericFolderItemEnum -MailboxName gscales@datarumble.com -FolderPath \Inbox -Credentials $creds -MaxCount 10 -FullDetails
If you want to recurse though Subfolders eg to show the top 5 messages for the Inbox and its subfolders use something like 

Invoke-GenericFolderItemEnum -MailboxName gscales@datarumble.com -FolderPath \Inbox -Credentials $creds -MaxCount 5 -Recurse | Select FolderPath,Subject
Like the Generic EWS Folder Object I've added some properties and methods that I've found useful. The first is the FolderPath which is useful when doing a recursion and reporting and I've also added one script method on objects called DownloadAttachments that will allow you to download all the attachments on an Item to the directory you pass in. eg  To download the attachments from the last item in the Inbox use


Updating your tools

Unlike like a normal spanner which is literally forged in steel this tool can be updated as you go but adding in extra properties or methods that you find useful as you go. If you find the script useful and what to share any of you updates please submit them to the GitHub repo https://github.com/gscales/Powershell-Scripts/issues