Friday, April 12, 2019

The new Mail.ReadBasic permission for the Microsoft Graph API and how to put it to use

Microsoft have just released the Mail.ReadBasic permission into beta for the Microsoft Graph endpoint https://developer.microsoft.com/en-us/graph/blogs/new-basic-read-access-to-a-users-mailbox/ which is a much needed addition that allows the creation of automation and apps that can just access messages at a more meta information level without having access to the Body or Attachments. Privacy and security are always pressing issues especially around email so this can turn down the privacy concerns while also reduce the security concerns of giving full access to content. 

Lets look at one use case for this new permission which is getting the Message Headers from a Message that has arrived in a users Mailbox that you suspect might be spam but you want the header information to do some analysis. Normally if you wanted to build an app to automate this you would have to at least assign Mail.Read which would give full access to all email content in a Mailbox (either delegated or every mailbox in a tenant in the case of Application permissions). This new grant allows us to just to get the Meta information like TO/From and all the first class properties which now includes the InternetMessageHeaders https://docs.microsoft.com/en-us/graph/api/resources/internetmessageheader?view=graph-rest-1.0

All you need to get going with using this is an application registration with just the Mail.ReadBasic permission assigned (for delegate access you would also need access to their underlying Exchange Folder or Mailbox your going to be querying via the normal Exchange DACL mechanisms).

I've put together a simple script that uses the ADAL for authentication and you can then search for a message based on the Internet MessageId and it will retrieve and then process the antipsam headers so you can look at DKIM,SPF and DMAC information just with this permission grant.


An example of this in use say if we are looking at the last 60 minutes of our trace logs for messages that where FilteredAsSpam (meaning the message ended up in the Junk Mail folder in the Mailbox)


We can take that MessageId and feed it the script cmdlet and get



A few things that are missing for this at the moment are to be really useful it needs to be an Application permission which I believe is coming. The other thing is you really need to be able to enumerate the Folder Name which this restricted at the moment and the ItemClass should be a first class property as you need it to determine the different types of emails you might be detail with.

Thursday, March 28, 2019

How to log EWS Traces to a file in PowerShell

If your using the EWS Managed API in your PowerShell scripts and you need to do some extended debugging to work out why a script isn't working the way you expect in certain environments you can do this by using Tracing as described in https://docs.microsoft.com/en-us/previous-versions/office/developer/exchange-server-2010/dd633676(v=exchg.80) . What this does once it is enabled is it outputs all the requests and responses that are sent to and from the Exchange server so you can see exactly what is taking place and potentially more information on particular errors that are occurring.  So in a EWS Managed API script to enable this you just need to set the TraceEnabled property on the ExchangeService object to true eg

$server.TraceEnabled = $true

And you will then start seeing traces like the following in the console



A much cleaner way of capturing these traces is to configure the EWS Managed API to use a separate log file to log them to a file so you can review them later. To do this it requires that you create a class that implements an Interface of ITraceListener https://github.com/OfficeDev/ews-managed-api/blob/70bde052e5f84b6fee3a678d2db5335dc2d72fc3/Interfaces/ITraceListener.cs .  In C# this a pretty trivial thing to do but in PowerShell its a little more complicated. However using Add-Type in PowerShell gives you the ability to simply define your own custom class that implements the interface and then compile this on the go which then makes it available in your PS Session. The basic steps are

  • You need to define an class that implements the interface (through inheritance) and the methods defined in that interface in this case it only has one called Trace
  • Define your own code to perform the underlying logging in my example its a simple one liner that will append the Tracemessage to a File the path of which is held in the Public Property I've defined in my class 
  • Use Add-Type to compile the class and make it available in your PS Session
  • Create a Instance of the Class you just defined eg here's a function to do it
eg
function TraceHandler(){
$sourceCode = @"
    public class ewsTraceListener : Microsoft.Exchange.WebServices.Data.ITraceListener
    {
        public System.String LogFile {get;set;}
        public void Trace(System.String traceType, System.String traceMessage)
        {
            System.IO.File.AppendAllText(this.LogFile, traceMessage);
        }
    }
"@    

    Add-Type -TypeDefinition $sourceCode -Language CSharp -ReferencedAssemblies $Script:EWSDLL
    $TraceListener = New-Object ewsTraceListener
   return $TraceListener


}

Then in your PS Code just use the Instance (Object) of the Class you just created (first setting the LogFile property to path of the File you want to log to) eg

        $service.TraceEnabled = $true
        $TraceHandlerObj = TraceHandler
        $TraceHandlerObj .LogFile = "c:\Tracing\$MailboxName.log"
        $service.TraceListener = $TraceHandlerObj 

Friday, March 08, 2019

Microsoft Teams Private Chat History Addin for Outlook

Being somebody who is transitioning across from Skype for Business to Teams one of things I missed the most (and found the most frustrating) is the lack of the ability in Outlook and OWA to view the conversation history from Online meetings and private chats in Microsoft Teams. This is especially frustrating when you have an external meeting and your sent an IM that contains some vital information for what you need to do. This information is tracked in your mailbox for compliance reasons in the Teams Chat folder but this folder is hidden so it not accessible to the clients and must be extracted by other means eg. Most people seem to point to doing a compliance search if you need this data https://docs.microsoft.com/en-us/microsoftteams/security-compliance-overview .

Given that the information is in my mailbox and there shouldn't be any privacy concern around accessing it I looked at a few ways of getting access to these TeamsChat messages in OWA and Outlook the first way was using a SearchFolder. This did kind of work but because of a few quirks that the Hidden folder caused was only usable when using Outlook in online mode (which isn't very usable). The next thing I did was look a using an Addin which worked surprising well and was relatively easy to implement. Here is what it looks likes in action all you need to do is find an Email from the user you want to view the Teams chat messages from and then a query will be executed to find the last 100 Chat messages from that user using the Outlook REST endpoint eg



That constructs a query that looks like the following to Outlook REST endpoint


https://outlook.office.com/api/v2.0/me/MailFolders/AllItems/messages?$OrderyBy=ReceivedDateTime Desc&$Top=30&$Select=ReceivedDateTime,bodyPreview,webLink&$filter=SingleValueExtendedProperties/Any(ep: ep/PropertyId eq 'String 0x001a' and ep/Value eq 'IPM.SkypeTeams.Message') and SingleValueExtendedProperties/Any(ep: ep/PropertyId eq 'String 0x5D01' and ep/Value eq 'e5tmp5@datarumble.com')

To break this down a bit first this gets the first 30 messages from the AllItems Search Folder sorted by the ReceivedDateTime

 https://outlook.office.com/api/v2.0/me/MailFolders/AllItems/messages?$OrderyBy=ReceivedDateTime desc&$Top=30
Next this selects the properties we are going to use the table to display, I used body preview because for IM's that generally don't have subjects so getting the body preview text is generally good enough to shown the whole message. But if the message is longer the link is provided which will open up in a new OWA windows using the weblink property which contains a full path to open the Item. One useful things about opening the message this way is you can then click replay and continue a message from IM in email with the body context from the IM (I know this will really erk some Teams people but i think it pretty cool and has proven useful for me).

$Select=ReceivedDateTime,bodyPreview,weblink

Next this is the filter that is applied so it only returns the Teams chat messages (or those messages that have an ItemClass of IPM.SkypeTeams.Message and are from the sender associated with the Message you activate the Addin on. I used the Extended property definition for both of these because firstly there is no equivalent property and for the From address if you used orderby and the a from filter like  and from/emailAddress/address eq 'e5tmp5@datarumble.com' there's a bug that the messages won't sort by the date so you always get the old messages first. Using the extended property fixed that issue but its a little weird.

 $filter=SingleValueExtendedProperties/Any(ep: ep/PropertyId eq 'String 0x001a' and ep/Value eq 'IPM.SkypeTeams.Message') and and SingleValueExtendedProperties/Any(ep: ep/PropertyId eq 'String 0x5D01' and ep/Value eq 'e5tmp5@datarumble.com') One thing I did find after using this for a while is that it didn't work when I got a notification from teams like the following


Because the above notification message came from noreply@email.teams.microsoft.com it couldn't be used in the above query. Looking at the notification message unfortunately there wasn't any other properties that did contain the email address but the full DisplayName of the user was used in the email's displayName so as a quick workaround for these I made use of EWS's resolvename operation to resolve the displayName to an email address and then I could use the Addin even on the notification messages to see the private chat message that was sent to me within OWA without needing to open the Teams app (which if you have teams account in  multiple tenants can be a real pain). So this one turned into a real productivity enhancer for me. (A quick note is that this will only get the Private Chat messages from the user not the Channel Messages).

Want to give it a try yourself ?

I've hosted the files on my GitHub pages so its easy to test (if you like it clone it and host it somewhere else). But all you need to do is add it as a custom addin (if your allowed to) using the
URL-

  https://gscales.github.io/TeamsChatHistory/TeamsChatHistory.xml



The GitHub repository for the Addin can be found here https://github.com/gscales/TeamsChatHistoryOWAAddIn

Tuesday, February 19, 2019

How to unsubscribe from a Mailing list using the Graph API

One of the features that is currently in beta in the Microsoft Graph API is an operation that will let you unsubscribe from any mailing list that supports the List-Unsubscribe header in a message that complies with RFC-2369 (https://docs.microsoft.com/en-us/graph/api/message-unsubscribe?view=graph-rest-beta).

This can have a number of uses one that does come to mind is when you have staff that are leaving the company (or even taking an extended break where they won't be reading their email) and they have signed up to quite of number of mailing lists. As part of your deprovisioning process you can include a script that will unsubscribe from the emails in a Mailbox before you remove the account instead of deleting and hoping the NDR's do it for you.

The RFC state the following for List-Unsubscribe

3.2. List-Unsubscribe

   The List-Unsubscribe field describes the command (preferably using
   mail) to directly unsubscribe the user (removing them from the list).

   Examples:

     List-Unsubscribe: 
     List-Unsubscribe: (Use this command to get off the list)
         
     List-Unsubscribe: 
     List-Unsubscribe: ,
         
Notably the word preferably is used in the RFC which means that from a implementation standpoint you don't have to have an unsubscribe email address to comply with this RFC. One example of this is  LinkedIn which only has URL's for the unsubscribe which requires you click a checkbox etc to unsubscribe which does nullify the usefulness of this somewhat.

To use this operation is pretty simple all you need is the Id of the mail you want to unsubscribe to and then you do a post on the unsubscribe nav

/users('user@domain')/messages/{id}/unsubscribe

I've created a simple ADAL graph script that gets a unique list of un-subscribable  email for the last 1000 emails in a Mailbox and then runs the unsubscribe method on those emails and posted It https://github.com/gscales/Powershell-Scripts/blob/master/Unsubscribe-Emails.ps1.

 I've also added support for  this into my Exch-Rest module which is available from the PowerShell Gallery and GitHub

To show the unsubscribe information for the last 100 messages in the Inbox use

Get-EXRWellKnownFolderItems -MailboxName gscales@datarumble.com -WellKnownFolder Inbox -MessageCount 100 -ReturnUnsubscribeData | select Subject,unsub* | fl

To process all the email from the last 7 days and unsubscribe for that using something like

        $UnSubribeHash = @{}
        -Filter ("receivedDateTime ge " + [DateTime]::Now.AddDays(-7).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ")) -ReturnUnsubscribeData | where-object {$_.unsubscribeEnabled -eq $True} | ForEach-Object{
            $Message = $_
            if($Message.unsubscribeEnabled){
                foreach($Entry in $Message.unsubscribeData){
                    if($Entry.contains("mailto:")){
                        if(!$UnSubribeHash.ContainsKey($Entry))
                        {
                            Invoke-EXRUnsubscribeEmail -ItemRESTURI $Message.ItemRESTURI
                        } 
                    }
                }
            }
        }





Friday, January 11, 2019

Create a Microsoft Teams Group Calendar tab application using the Graph API and FullCalendar JavaScript library

Group calendars have always been one of the big asks for in any group collaboration programs back from Lotus Notes to Microsoft Exchange and now Microsoft Teams. There are a few ways of getting a Group calendar working in Teams, one is hosting the OWA web apps see or some other people advocate using a SharePoint calendar and hosting that similarly.

Here is a different method you can use by taking advantage of being able to call the Graph API in a  Tab application.The getSchedule Graph action (still currently in beta) allows you to query up to 62 days of Freebusy information on up to 100 calendars in a single call this makes it a good option for this type of application. So as long as users can view each others calendars or have detailed freebusy permissions the action should return the level of detail required for a Group calendar.   The other thing you can do with the Graph API is get the users photo and build a nice legend for the Group calendar also. To make this visually appealing you need a good calendar display library which there has been a few over the years but a recent one that is really nice is FullCalendar https://fullcalendar.io/ . It ticks all the boxes it looks great, its easy to use and its free to use. To make the calendar appointments from graph appear in the calendar is as easy as building and array from the return JSON from the graph and throwing in a little random color code to break this up and then stitching in the user photos as they are returned asynchronously from the server to build the legend.


 Here are some screenshots of the Tab in action using Graph data




    Daily view


Weekly view


List view


I've put together a separate GitHub repository for all the code that is required for this app https://github.com/gscales/TeamsGroupCalendar and put some detailed install instruction in the Readme in Github.

I'm currently looking for work either contract or fulltime so if you need a creative developer with lots of energy to write C#,JS, NodeJS, Azure or Lambda functions, Messaging DevOps or PowerShell scripts then please contact me  at gscales@msgdevelop.com

Tuesday, January 08, 2019

Converting Folder and ItemIds from the Exchange Management Shell and Audit Log entries using PowerShell and the Graph API in Exchange Online

First a little news about Exchange Identifiers that you may have missed (its not often that something like this changes so its rather exciting)

When you access an item in an Exchange Mailbox store whether its OnPrem or in the Cloud you use the Identifier of the particular item which will vary across whatever API your using. Eg

MAPI - PR_EntryId eg NameSpace.GetItemFromID(EntryId)
EWS -  EWSId eg EmailMessage.Bind(service,ewsid)
Rest -   RestId  eg https://graph.microsoft.com/v1.0/me/messages('restid')
The advice over the years has always been its not a good idea to store these Id's in something like a database because they change whenever and Item is moved. Eg if an Item is moved from the Inbox to a Subfolder in the Inbox it will received a different Id so whatever you have stored in your database suddenly becomes invalid and its not easy to reconcile this. However a new feature that has appeared in Exchange Online in Beta with the Graph API is immutableId's see https://docs.microsoft.com/en-us/graph/outlook-immutable-id the idea behind this is that this Id doesn't change regardless of which folder the item is moved to (or even if its deleted). While it still in Beta at the moment this is a good feature to use going forward if your building synchronization code. Along with immutableId's an operation to Translate Id's between the EntryId, EWS and REST formats is now available in beta in the Graph which is great if your looking to Migrate your MAPI or EWS apps to use the Graph API https://github.com/microsoftgraph/microsoft-graph-docs/blob/master/api-reference/beta/api/user-translateexchangeids.md 

As Audit records are a hot topic of discussion this week with this post from Microsoft another Identifier format you see when using the Exchange Management Shell cmdlets like Get-MailboxFolderStatics is something like



or in an ItemId in a  AuditLog Record like



With these Id's there are just a base64 encoded version of the EntrydId with a leading and trailing byte. So to get back to the Hex version of the Entryid you might be familiar with from a Mapi Editor you can use something like the following



$HexEntryId = [System.BitConverter]::ToString([Convert]::FromBase64String($_.FolderId.ToString())).Replace("-","").Substring(2)  
$HexEntryId =  $HexEntryId.SubString(0,($HexEntryId.Length-2))

This would turn something like

RgAAAAC+HN09lgYnSJDz3kt9375JBwB1EEf9GOowTZ1AsUKLrCDQAAAAAAENAAB1EEf9GOowTZ1AsUKLrCDQAALbJe1qAAAP

Into

00000000BE1CDD3D9606274890F3DE4B7DDFBE490700751047FD18EA304D9D40B1428BAC20D000000000010D0000751047FD18EA304D9D40B1428BAC20D00002DB25ED6A0000

Just having the Id in whatever format isn't much good unless you can do something with it, so I've created a simple Graph script that uses the new user-translateexchangeids.md operation to allow you to translate this Id into an Id that would be useable in other Graph requests. I've create a basic ADAL script version an posted it here on my GitHub https://github.com/gscales/Powershell-Scripts/blob/master/translateEI.ps1

A quick Demo of it in use eg Translate a RestId into an EntryId


Invoke-TranslateExchangeIds -SourceId "AQMkADczNDE4YWE..." -SourceFormat restid -TargetFormat entryid
By default the operation returns a urlsafe base64 encoded results (with padding) so in the script I decode this to the HexEntryId which I find the most useful.

I've also cater for allowing you to post a HexEntryId and the script will automatically encode that for the operations eg


Invoke-TranslateExchangeIds -SourceHexId "00000000BE1CDD3D9606274890F3DE4B7DDFBE49..." -SourceFormat entryid -TargetFormat restid
And it also caters for the encoded EMS format and will strip the extra bytes and covert that eg

Invoke-TranslateExchangeIds -SourceEMSId  $_.FolderId.ToString() -SourceFormat entryid -TargetFormat restid
I've also added this to my Exch-Rest module which is available from the PowerShell Gallery and GitHub which is useful if you want to do some following type things. eg if you wanted to bind to the folder in question you could use


$folderId = Invoke-EXRTranslateExchangeIds -SourceEMSId  $_.FolderId.ToString() -SourceFormat entryid -TargetFormat restid
Get-EXRFolderFromId -FolderId $folderId
Need help with anything I've talked about in this post or need somebody to write C#,JS, NodeJS, Azure or Lambda functions, Messaging DevOps or PowerShell scripts then I'm available now for freelance/contract or fulltime work so please drop me an Email at gscales@msgdevelop.com

Friday, January 04, 2019

Using the Skype for Business UCWA API in a Microsoft Teams Tab application to show the Skype Conversation history

One of the things you maybe considering in the new year is migrating from Skype for Business to Microsoft Teams. In this post I'm going to demonstrate how you can use the UCWA api (which is the REST API you can use to talk to a Skype for Business server either in Office365 or OnPrem) to access Skype for Business from within the Teams Client via a Teams Tab application. (For those unacquainted with UCWA this the API that is used to Access Skype within OWA).
Why would you want to do this ? its one way of easing migration friction by providing a different level of interoperability (outside of using both clients) and also a way of adding functionality into the Teams client that isn't there currently.  In this post I'm going to look at showing the users Skype conversation history, while this information is also stored in a users Mailbox and also accessible via the Graph API, in this app I'm going to use the UCWA API to access the conversation logs via the Skype for Business Online servers and also the conversation transcripts. What you end up with is a Teams  tab that looks like the following



and the Transcripts like (this is POC so mind the formatting)



Using UCWA from a Teams Tab Application

There isn't much difference between using the UCWA API in a Teams Tab application then  from using it in any other application, however UCWA does present some challenges around authentication because of the way the discovery process works.  For a quick recap for those unacquainted please read https://docs.microsoft.com/en-us/skype-sdk/ucwa/developingucwaapplicationsforsfbonline . As part of that process you need to get an AccessToken to make a discovery request to find the SK4B pool server to use and then get another AccessToken for the pool server. So when using the Teams Tab Silent authentication flow you need to execute this twice (which is different and more time consuming then say a normal Graph type application)

Getting the Conversation History in UCWA

Once you have logged into UCWA you need to configured the session to enabled the conversation history as its disabled by default.  This involves doing a Put request against the application resource. with the if/match header set to the ETag. You then need to acknowledge the event this will generate and once that is done your UCWA session will be ready to go, you then just need to query the communication resource to get the links required to access the ConversationLogs from the server. In the sample app I'm only accessing the last 50 items from the server as this is only a POC anyway. When it comes to access the conversation transcripts this requires a Batch request to make it efficient (the max batch size in SK4B in 100) so using a page size of 50 keeps this all working okay. A brief overview of the requests required to access the Conversation history.

  • 1 Get request to get the Conversation Logs which is a list with a link to each of the Conversation Entries
  • Batch Get Request for each of the Conversation Entries which gives back a detail history of each conversation (minus the actual Transcript of the conversation but you do get limited Message preview).
  • If you want the full conversation transcript you use the link from the conversation history to access the Transcript. (in the sample when you click the transcript Cell in the Table it makes this request to the SK4B server to get the Transcript and presents that in a separate Div on the page),
Installing and using this Tab Application 

Like any Teams Tab application it must be hosted somewhere, I'm hosting it out of my GitHub site so the configuration file located in https://github.com/gscales/gscales.github.io/blob/master/TeamsUCWA/app/Config/appconfig.js has the following configuration to ensure it point to the hosted location

const getConfig = () => {
   var config = {
        clientId : "eed5c282-249f-46f3-9e18-bde1d0091716",
        redirectUri : "/TeamsUCWA/app/silent-end.html",
        authwindow :  "/TeamsUCWA/app/auth.html",
 hostRoot: "https://gscales.github.io",
   };
   return config;
}

Also the manifest file https://github.com/gscales/gscales.github.io/blob/master/TeamsUCWA/TabPackage/manifest.json  has setting that point to hosted that need to be changed if its hosted elsewhere (just search and replace gscales.github.io)

Application Registration 

To use the UCWA API you need to create an application registration with the following oAuth grants



the applicationId for this registration should then be used to replace the clientid in the appconfig.js . The application registration should use the silent-end.html page as the redirect for authentication. Then the last thing you need to do is make sure that the ApplicationId has been consented to in your Organization eg 


https://login.microsoftonline.com/common/adminconsent?client_id=08401c36-6179-4bbe-9bcc-d34ff51d828f
  
Side Loading - To use custom tab applications you first need to enable side loading of Apps in the Office365 Admin portal ref .The important part is  "Sideloading is how you add an app to Teams by uploading a zip file directly to a team. Sideloading lets you test an app as it's being developed. It also lets you build an app for internal use only and share it with your team without submitting it to the Teams app catalog in the Office Store. "

As this is a custom application you need to use the "upload a custom app" link which is available when you click ManageTeam-Apps tab see

(Note if you don't see  the "upload a custom app" check that you have side loading of apps enabled in your tenant config)

What you upload here is a Zip file containing your manifest file and two images that you manifest refers to for eg
{
   "icons": {
    "outline": "Outline32.png",
    "color": "Colour192.png"
  },

For this sample this is located in https://github.com/gscales/gscales.github.io/blob/master/TeamsUCWA/TabPackage/manifest.json

All the code for this post is located in GitHub https://github.com/gscales/gscales.github.io/tree/master/TeamsUCWA

Need help with anything I've talked about in this post or need somebody to write C#,JS, NodeJS, Azure or Lambda functions or PowerShell scripts then I'm available now for freelance/contract or fulltime work so drop me an Email at gscales@msgdevelop.com