Friday, October 19, 2018

Using the LAPFID (Last Active Parent FolderId) in EWS and the Graph API in PowerShell when reporting on Deleted Items

A little background

The LAPFID property is an extended property that gets set on an Exchange Store Item when its deleted (any type of delete soft or hard) that is the enabler for the original folder item recovery feature that rolled out to both Exchange Online and Exchange OnPrem (2016) last year. If you where to look at this property in a MAPI editor it would look like the following



To understand what that property Id value represents you need to first understand a little bit more about the Folder EntryId format (PR_EntryId) that exchange uses which is documented in this Exchange Protocol Document https://msdn.microsoft.com/en-us/library/ee217297(v=exchg.80).aspx . A different visual representation of this with the individual components highlighted would look something like this

Hopefully from this you can see that the LAPFID is comprised of the DatabaseGUID and GlobalCounter constituents of the FolderEntryId. To make sense of the LAPFID you need to resolve it back to the actual Mailbox Folder that it represents. A couple of ways you could go about its is to reconstruct the PR_EntryId using the values from the LAPFID and the header values from another folder in the Mailbox which should be the same and then convert that to a EWSId using ConvertId and then Bind to the Id that is returned. The other way which is the method I've used is to enumerate all the Folders in the Mailbox (and Archive store if it exists) and then get the DatabaseGuid + GlobalCounter combinations from the PR_EntryID of those folders, stick this into a Hashtable and then you can look up that for each item to do the resolution process. One issue is that the folder in question in the LAPFID may have been deleted itself which would make both those processes fail.

Getting the LastActiveParentEntryId property in EWS and the Graph API

To get this property value in EWS you need to request the extended property like the following in the EWS Managed API

$LastActiveParentEntryId = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x348a, "Binary")

or if your using the Graph API your request should look something like the following

$expand=SingleValueExtendedProperties($filter=Id eq 'Binary 0x348A'))

Putting them to work in a Script

Apart from the obvious thing of being able to restore items to their original location the LAPFID property can have a variety of reporting uses. I've put together a small module to allow easy access to this property along with some expansion to strongly typed properties so it will show the original folderpath and also the original EWS Folder (if you want to do the restore). I've also done the same for the Graph API in my Exch-Rest module where I've added some cmdlets to view all the deleted and recoverableItems folders and to retrieved this property and do the expansion of the Folder.

Some modifications to the base EWS Script

Because its now more important then ever to stop using Basic authentication for anything see the escalation https://blogs.technet.microsoft.com/exchange/2018/10/17/disabling-basic-authentication-in-exchange-online-public-preview-now-available/  I've made changes to the base script I use for EWS and I've now defaulted the Authentication to use oAuth. While if you are using onPrem you may still need to use basic Auth I have provided provision for doing this as well but will require that you use the -basicAuth switch. I also have some different options for oAuth eg if you want to use the ADAL.dll to generate the access token the front-end cmdlet will take the AccessToken via the -AccessToken parameter if nothing is passed in it will use some generic PowerShell methods to generate an Access Token.

Some Examples in Action

The EWS Module has a cmdlet called Get-LAPFIDItemsFromFolder which has three switch's that can be used to access either the DeletedItemsFolder, RecoverableItemsDeletions and  RecoverableItemsPurges. Or you can also use the -FolderPath parameter and pass in a FolderPath if you really like.

eg to get the items form the dumpster's deletions folder use something like


By default only the Primary Mailbox of the user you enter is checked but if you want to also check the Archive you can use the Archive switch eg

Get-LAPFIDItemsFromFolder -MailboxName jcool@datarumble.com -recoverableItemsDeletions -Archive | select LastActiveParentfolderPath,Subject,Size



Or maybe you want to see how many items from each LAPFID folder in the DeletedItems folder

Get-LAPFIDItemsFromFolder -MailboxName gscales@datarumble.com -deletedItems | Group-Object LastActiveParentfolderPath


Or by RetentionDate

Get-LAPFIDItemsFromFolder -MailboxName gscales@datarumble.com -deletedItems | Group-Object RetentionDate | Sort-Object Count -Descending


Show items from a particular LAPFID

Get-LAPFIDItemsFromFolder -MailboxName gscales@datarumble.com -deletedItems | Where-Object {$_.LastActiveParentFolderPath -eq "\Junk E-mail"} | select DateTimeReceived,Subject,Size


To Restore an item to its original folder you just use the normal EWS Move operation, because I'm using the EWS Managed API there is a Move method on Each object that can be called so you just need to pass in the folderId of the folder you want to move it to. In the case the original folder you can just pass in the expanded value. eg if I get a collection of items like

$items =  Get-LAPFIDItemsFromFolder -MailboxName gscales@datarumble.com -deletedItems

and you want to restore the first item (which would the also the last item deleted all you need is)

$items[0].move($items[0].LastActiveParentFolder.Id)

this would move the items back to its LastActiveParentFolder value.

Exch-REST and the Graph API

The Graph API has a few restrictions around Deleted Items when compared against EWS eg because the Deleted Items access method uses the Mail Endpoint you will only be able to get Items with a ItemClass (or Subclass) of IPM.Note. So while other things like contacts and appointments maybe in these folder these items won't show in the API. Simular with Folders if the folder doesn't have a Class or Subclass of IPF.Note it won't show. And lastly you can't currently access the Archive using the Graph API. (but apart from that)

In the Exch-Rest module there a 3 cmdlets that can be used to return the information


Because each of the cmdlets in this module build on each other you can do some interesting things like process the Antispam header to get the DMARC and DKIM information and show that along with the LAPFID eg

 Get-EXRDeletedItems -ReturnLastActiveParentFolderPath  -ReturnInternetMessageHeaders -ProcessAntiSPAMHeaders | select LastActiveParentFolderPath,DMARC,DKIM


Or the Sentiment property to see if people are deleting negative email eg

Get-EXRDeletedItems -ReturnLastActiveParentFolderPath  -ReturnSentiment | select LastActiveParentFolderPath,Sentiment,Subject | where-object {$_.Sentiment -ne $null}



The EWS PowerShell Module for this can be found on GitHub Here https://github.com/gscales/Powershell-Scripts/blob/master/LAPFIDModule.ps1 , the Exch-REST Module is available from the PowerShell Gallery https://www.powershellgallery.com/packages/Exch-Rest and GitHub https://github.com/gscales/Exch-Rest

Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365,Microsoft Teams, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com(nothing too big or small).




Wednesday, October 10, 2018

Accessing Public Folder data from a Microsoft Teams Tab application using EWS

For a long time now Public Folders have been a good way to share information and have threaded conversations across organizations. However time and technology have surpassed them somewhat of late, if your using Office365 there was first the introduction of Unified Groups and now Microsoft Teams has come to maturity both of which offer a better user and technical solution to needs Public Folders fulfil. The benefits of these newer technologies is they allow you to do more in the same context while not giving up on the existing features that Public Folders may have been giving you (outside of access in Outlook). But where the rubber meets the road at the coal face its not always as easy to migrate from one solution to another, so in this post I'm going to look at how you can access data (email etc) in an Exchange Public folder from within a Team's Tab application. This type of approach may give you some flexibility and options while your dealing with a Migration between the two or just during that tricky cutover period where you don't have everyone onboard at once.


What it looks like


the Text box is a simple search bar that takes a KQL query so by default it will just return everything but if you entering a query and hit the refresh button you will get filtered results eg





Technical Challenges  

Over the past couple of months I've posted a few Teams tab applications that expose Exchange data using the Graph API as the interface into the Exchange Store. Public Folders aren't yet accessible via the Graph API so to access these you need to use Exchange Web Services. While this isn't too difficult in itself because EWS doesn't support CORS it means you can't write JavaScript code that will run in the client browser to access EWS directly.

Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTPheaders to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin. In the context of a Teams tab application that wants to access Public folders via EWS the source domain will be where the teams tab application pages are being hosted and the target would be EWS outlook.office365.com, because the server where EWS is hosted doesn't support CORS the browser will block this request because the correct headers aren't returned when the request is pre-flighted (http Options request)
To mitigate against CORS you can use a Proxy like this node.js server https://www.npmjs.com/package/cors-anywhere then if your writing your client side code in something like Angular you can proxy your requests via this. Another way which is the one I've chosen is to write all the EWS logic and EWS calls in node.js and then create a simple Rest API for the Node server that can be called from the client side code that runs in Teams. This has a few benefits as there are multiple requests to get the public Folder and correct routing header initially. With this method you can run all these from a cloud hosted server that should have lower latency then the client which should mean better performance and also the service itself can be reused from other applications.

Node.js app

The node.js server is a relatively straight forward app that uses Express and has one route that become the REST endpoint that the Teams Tab application will call. I've used a JavaScript port somebody did of the EWS Managed API https://github.com/gautamsi/ews-javascript-api  which I found was reasonably easy to use and made dealing with the EWS side of the code a lot faster because I could port over existing C# code. One of the least fun things to do when dealing with Public Folder and EWS is to workout the correct routing headers https://docs.microsoft.com/en-us/exchange/client-developer/exchange-web-services/public-folder-access-with-ews-in-exchange but I have some code that will handle this and also finding the Public folder from a path using multiple shallow traversals. To allow paging past 1000 object (or I made the default page size about 100 to maintain reasonable performance) the rest endpoint support calling it again with an EWS folderId as to avoid the discover process a second time. Other then that the EWS code is pretty run of the mill findItems, folderfinds and bind.

Teams Tab application

Like the other Teams tab applications I wrote this use the Teams Tab Silent authentication method but instead of getting a token for the Graph Endpoint its gets it for the EWS endpoint outlook.office65.com. For displaying the email data I've used the tabulator JavaScript library which has a few nice feature I like.  Each of the email is clickable and will open any of the items you click up in a new tab in OWA eg


Using this app

Unlike the previous tab apps I've published which could just be used straight from GitHub this one requires some technical understanding. Firstly you will need to host the node.js server somewhere eg Azure is best place as that gets you as close to the Exchange servers that will be serving the content. Once you have this hosted all the configuration for the app is held in the appconfig.js file https://github.com/gscales/gscales.github.io/blob/master/TeamsPublicFolder/app/Config/appconfig.js


const getConfig = () => {
   var config = {
        clientId : "2417973f-0680-4ea0-a844-e6d414cee4d4",
        redirectUri : "/TeamsPublicFolder/app/silent-end.html",
        authwindow :  "/TeamsPublicFolder/app/auth.html",
 hostRoot: "https://gscales.github.io",
 folderpath: "\\Childfolder\\childfolder",
 ewsproxy: "https://459e25ee.ngrok.io/FolderItems",
   };
   return config;
}

clientId is for the application registration you should create for this app, to access EWS you only need to the following 1 permission


folderpath is the path to the public folder (because this is a js file there is one more \ the normal to escape the path \.

ewsproxy is the REST endpoint of node.js server in this example I was hosted in locally on my pc and then exposting that endpoint using ngrok which is a great tool for prototyping.

You also need to host the Tab application pages somewhere and modify the manifest to indicate where the location is.

other this this the normal

Side Loading - To use custom tab applications you first need to enable side loading of Apps in the Office365 Admin portal ref .The important part is  "Sideloading is how you add an app to Teams by uploading a zip file directly to a team. Sideloading lets you test an app as it's being developed. It also lets you build an app for internal use only and share it with your team without submitting it to the Teams app catalog in the Office Store. "

As this is a custom application you need to use the "upload a custom app" link which is available when you click ManageTeam-Apps tab see

(Note if you don't see  the "upload a custom app" check that you have side loading of apps enabled in your tenant config)

What you upload here is a Zip file containing your manifest file and two images that you manifest refers to for eg
{
   "icons": {
    "outline": "Outline32.png",
    "color": "Colour192.png"
  },

For this sample this is located in https://github.com/gscales/gscales.github.io/tree/master/TeamsPublicFolder/TabPackage

These icons are what is used in the UI to represent your application.

 The files for the tab application can be found on GitHub https://github.com/gscales/gscales.github.io/tree/master/TeamsPublicFolder

The Node.Js solution for the EWS proxy can be found in this repo https://github.com/gscales/TeamsPublicFolder-NodeEWSProxy

There are some todo's that still need to be done eg the pagination is a little clunkily as there is a differences between paging between what is returned (eg in EWS you page back item in lot of 1 to 1000) and paging between all the items in a Public Folder so there is separate footer button to get the next 100 items (from the server) and separate buttons to page the results. Most of these are relativity easy to solve if your interested in using this considering hiring me to complete it and share it with others :).

Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365,Microsoft Teams, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com(nothing too big or small).


Monday, September 24, 2018

Microsoft Teams Tab Application that uses the Graph API and Exchange FreeBusy Information

In my last post I went over the new Graph function called getSchedule and how this can be used to get the FreeBusy information for a User (or Meeting Room, Resource Mailbox etc) and display that as a table like the schedule assistant in Outlook.

In this post I'll show you how you can do the same thing in a Teams tab application which will give you an output something like the following


The steps this app takes to display this is firstly

Step 1.

Authentication - The app authenticates to the Microsoft Graph API using the javascript ADAL using the silent Authentication flow for tab applications which is explained in detail here .

Step 2

Get the Group Members - Once it has an Access-Token it then Gets the members of the current Team from the Graph by first getting the Id from the Context of the team currently in focus from the Teams Client SDK and then making a request to the Groups Endpoint https://graph.microsoft.com/v1.0/groups/

Step3

Schedule Request - Make a schedule request of the Members of the Team (note this operation is only in Beta at the moment). This returns the FreeBusy information for the members of the team in 30 minute timeslots for the working hours period I'm displaying. Note the Timezone from the browser (or Teams Client) is used here. Incorrect Time zone of timezone differences are something that can affect the results of this operation.

Step4

Build the HTML to Display- Build the above FreeBusy table from the Results

Step5

Get the UesrPhotos -   Backfill the user photos asynchronously using the Graph photos endpoint.

That's it for the rundown on the functional side of how this Tab app works for more of a description on how Tab apps work in general have a look at my last post which goes a bit more in depth on this. I've rewritten most of the code from that post so its now ES6 and implements both promises and async functions in JavaScript which eliminants a lot of clutter,logic fog and call-backs code the plagued the original (if you still using IE none this will work but if your still using IE your probably using Microsoft comic chat).

I've also implemented a config file in

https://github.com/gscales/gscales.github.io/blob/master/TeamsFB/app/Config/appconfig.js

So anything that needed to be hardcoded in the app is in this file which is basically

        clientId : "2cff1b6d-82d4-4dc0-9ac4-7f35b06cc64a"  - this the Azure App registration which you should create your own of that will list

        redirectUri : "/TeamsFB/app/silent-end.html" - From the Azure App registration


        authwindow :  "/TeamsFB/app/auth.html" - For the Slient Auth functions

        hostRoot: "https://gscales.github.io" - For the relative references in the js files

this makes it easy if you want to change the host or use ngrok to host it, eg just change hostRoot, Azure RedirectURI to the ngrok address and you should be good to go.

Permissions for Azure App registration

Because this app accesses various graph endpoints different oauth Grants need to be allowed eg
  • Group members User.Read and Group.Read.All
  • FreeBusyInformation   Calendar.Read
  • UserPhotos User.ReadBasic.All
Installation and Pre-Req

Azure app registration - you need to have an app registration that is Authorised within you tenant (see my last post for more details)

Side Loading - To use custom tab applications you first need to enable side loading of Apps in the Office365 Admin portal ref .The important part is  "Sideloading is how you add an app to Teams by uploading a zip file directly to a team. Sideloading lets you test an app as it's being developed. It also lets you build an app for internal use only and share it with your team without submitting it to the Teams app catalog in the Office Store. "

As this is a custom application you need to use the "upload a custom app" link which is available when you click ManageTeam-Apps tab see



(Note if you don't see  the "upload a custom app" check that you have side loading of apps enabled in your tenant config)

What you upload here is a Zip file containing your manifest file and two images that you manifest refers to for eg
{
   "icons": {
    "outline": "Outline32.png",
    "color": "Colour192.png"
  },

For this sample this is located in https://github.com/gscales/gscales.github.io/tree/master/TeamsFB/TabPackage

These icons are what is used in the UI to represent your application.



All the code for this sample is available in GitHub here https://github.com/gscales/gscales.github.io/tree/master/TeamsFB

Its also hosted in GitHub pages if you have a development tenant and want to test it  https://gscales.github.io/TeamsFB (you will need to first use the admin consent URL listed above).







Friday, September 07, 2018

Getting FreeBusy information in the Graph API using the getSchedule action and putting it to use in PowerShell

One of the recent additions to the Graph API for Exchange in Office365 has been getSchedule action https://developer.microsoft.com/en-us/graph/docs/api-reference/beta/api/calendar_getschedule . What this action allows to do is get the FreeBusy information from one or more mailboxes in your Office365 environment. If your new to Exchange (or for those that have forgotten) FreeBusy information is essential what is used to produce the Schedule Assistant in Outlook and equivalent in OWA


The above is built by Outlook using the EWS GetUserAvaliblity operation which is what the getSchedule action in the Graph is the equivalent of. GetUserAvaliblity has been in use in Outlook since Exchange 2007 where we went from using Public Folders to hold the freebusy information in previous versions to the Public Folder free utopia we have today. In the preceding years features such as Suggested Meeting times have been added to this EWS operation this feature is available in the Graph via the Graph in the findmeeting action .

If you are using the Graph API
getSchedule is a  good a solution for solving one of the more frequently asked questions of how you can get the availability of multiple meeting rooms without needing to query every Meeting Rooms calendars (or a simular question for a group of users).

Limits

There are some limits to be aware of when querying FreeBusy data which is you can query up to 62 days worth of FreeBusy information (eg the period between the Start and EndTimes). The other limit you should adhere to is no more the 100 Mailboxes per request to keep below the throttling limits.

Making the query

The REST request you need to make to get the FreeBusy information should look something like the following

{
    "schedules":  [
                      "gscales@datarumble.com",
                      "mec@datarumble.com"
                  ],
    "endTime":  {
                    "dateTime":  "2018-09-08T10:21:49",
                    "timeZone":  "AUS Eastern Standard Time"
                },
    "startTime":  {
                      "dateTime":  "2018-09-07T10:21:49",
                      "timeZone":  "AUS Eastern Standard Time"
                  },
    "availabilityViewInterval":  15
}

Time zones are an issue that usually trips up a lot of people, especially if your dealing with mailboxes in multiple locations. Its important to note the timezone specified in the query only relates to the query period your using. The timezone that you get back in the results will depend on the request header Prefer: outlook.timezone in the Request. So if your setting the TimeZone in the query and your getting back a different unexpected TimeZone In the results you need to look at the Prefer: outlook.timezone that's being used (or the absence of that header will mean you will get UTC back).

The availabilityViewInterval affects the timeslots that come back  in the availabilityView generally you would use either 15,30 or 60 minutes but you can have a value as high as 1440 (which represents 1 day) and as low a 6 minutes. If you set the availabilityViewInterval to 1440 then just having one appointment in a day will mean that day will appear as Busy.

Results 

The results you get back from this action is first the availabilityView which depending on the interval you specified will be splices of availability between then Start and End Times you specified. eg



Also if you have the FreeBusy,Subject and location permission on the Target calendars (or higher) you will get an array of calendar appointments (as long as they aren't private) back for each Target Mailbox. eg


In EWS you also got back the HexEntryId of the appointment object meaning you could then retrieve more information for a specific Appointment without first doing a search which seems to be missing in the Graph operation at the moment. Lets hope that comes back as its an important function and the workaround is messy and will cause a lot of development pain for people 🙏🙏.

Using this in PowerShell

I've added the ability to use this action to my Exch-Rest PowerShell library which is what the above screen shots where produced using this module is available from the PowerShell Gallery https://www.powershellgallery.com/packages/Exch-Rest and GitHub https://github.com/gscales/Exch-Rest.

The Module has the following Cmdlet

Get-EXRSchedule

this takes an array of Mailboxes you want to run it against so to use it first create a blank array

$Mailboxes = @()

then populate that with the Mailboxes you want to run against eg

$Mailboxes += "gscales@datarumble.com"
$Mailboxes += "mec@datarumble.com"

Then run the cmdlet with the Start and EndTime and availability period you want by default these aren't need and it will return the freebusy information for the next 24 hours in 15 minutes intervals eg

Get-EXRSchedule -Mailboxes $mailboxes 

However usually you would want to run something like the following

$StartTime = [DateTime]::Parse((Get-Date).ToString("yyyy-MM-dd HH:00:00"))
$EndTime = [DateTime]::Parse((Get-Date).AddHours(6).ToString("yyyy-MM-dd HH:00:00"))
Get-EXRSchedule -Mailboxes $mailboxes -Start $StartTime -EndTime $Endtime -availabilityViewInterval 15

Which would get the freebusy information for the next 6 hours in 15 minute increments

The Module is useful for getting the data so lets look at a few fun ways you can use it. eg if you want to have the results stored in a collection you could export as CSV or Graph you could use something like the following

$rptCollection = @()
$StartTime = [DateTime]::Parse((Get-Date).ToString("yyyy-MM-dd HH:00:00"))
$EndTime = [DateTime]::Parse((Get-Date).AddHours(6).ToString("yyyy-MM-dd HH:00:00"))
$avResults = Get-EXRSchedule -Mailboxes $mailboxes -Start $StartTime -EndTime $Endtime -availabilityViewInterval 15
foreach($fbResult in $avResults){
 $CurrentTime = $StartTime 
 $fbResult.availabilityView.ToCharArray() | ForEach-Object{
  $rptobj = "" | select Time,User,FreeBusyStatus
  $rptobj.Time = $CurrentTime.ToString("HH:mm")
  $rptobj.User = $fbResult.scheduleId
  switch($_){
     0 {$rptobj.FreeBusyStatus = "Free"}
     1 {$rptobj.FreeBusyStatus = "Tentative"}
     2 {$rptobj.FreeBusyStatus = "Busy"}
     3 {$rptobj.FreeBusyStatus = "Out of Office"}
     4 {$rptobj.FreeBusyStatus = "Working Elsewhere"}
  }
   $rptCollection += $rptobj
  $CurrentTime = $CurrentTime.AddMinutes(15)
 }
}
return $rptCollection 


Or we can create a FreeBusy board using this information that would look like the following


Where I'm using both the avaiblitiyview data to create the Table and also the calendar details so when you hover over one of the busy table cells it will show the subject of that meeting. I've put the code up for the freebusy board here https://github.com/gscales/Powershell-Scripts/blob/master/fbBoardGraph.ps1 .

Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com(nothing too big or small).

Friday, August 24, 2018

Using Guest Access in PowerShell to access Unified Groups and Teams in Office365

This is a continuation somewhat of my previous post on using a generic oAuth script to access EWS but in this post I'm going to look at how you can authenticate as a Guest user to a tenant you have been granted Guest access to and then access the resources like a Unified Office365 Group or Teams( because Teams uses Groups) via the Graph API.

To Access another tenant as a Guest (considering that Guest Access is enabled and you have accepted the invitation and logged in at least once with that user) you need to generate an Access Token against the Target tenants endpoints. To do this you can discover what a particular tenants Authentication and Token endpoints are by making a request like the following

$RequestURL = "https://login.windows.net/{0}/.well-known/openid-configuration" -f $TargetDomain
$Response = Invoke-WebRequest -Uri  $RequestURL
$JsonResponse = ConvertFrom-Json  $Response.Content

This will return a result like the following


Out of this result the Authorization Endpoint and the Token_endpoint are what you need to make a successful authentication against the Target Tenant to get a guest access token.

For technical completeness the above step is not necessarily needed as the endpoint should also support you using the domain name as per https://docs.microsoft.com/en-us/azure/active-directory/develop/v1-protocols-oauth-code . But like many things in IT there are many ways of doing the same things.

I've posted up a script that first does the above the request and then will authenticate against the Target Tenant and return an Access Token that can then be used to access the Graph. I've created a test cmdlet called Test-GuestAuth that will enumerate the unified Groups the guest is a member of and get the first two conversation items  in each of the unified Groups, I've put the code up for this here

The script is on GitHub here https://github.com/gscales/Powershell-Scripts/blob/master/GenericOauthGuest.ps1

To use this script use something like the following

Test-GuestAuth -UserName gscales@datarumble.com -TargetDomain guesttenant.contos.com

For the clientId you would need to use in oAuth that clientId has to have been consented to by the Target Tenant admins, in the above script I've used the default Office clientId which should be enough to get access to Group conversation etc.

Friday, August 10, 2018

Dependency free Generic EWS oAuth PowerShell example for Office365

Microsoft recently posted an update that will affect those people who use EWS in applications and script against Office365 https://blogs.technet.microsoft.com/exchange/2018/07/03/upcoming-changes-to-exchange-web-services-ews-api-for-office-365 

While 2020 is a few years away what this means is that if you are using Basic Authentication in your EWS scripts or applications that on the 13th October 2020 your app will stop working. Given the amount of time you have and the changes required to support oAuth nobody should really be caught out by this but procrastination and people not understanding legacy applications will mean I'm sure this date won't pass without any infamy.

Within PowerShell scripts you have two options to generate the oAuth tokens you need to keep you script working. One is to use a dependency library like ADAL to do it which Ingo posted a really good write up for https://ingogegenwarth.wordpress.com/2018/08/02/ews-and-oauth the other is just create some of your own script code to do the Authentication and managed of the Tokens. As part of my Exch-Rest library because I wanted to make this dependency free I wrote my own routines for Getting and Renewing OAuth tokens needed which can also be used for EWS . So in this post I've separated those out and included them into a simple header script that can be used to do oAuth against Office365.

EWS Managed API and oAuth

If you using the EWS Managed API in your scripts which the majority of people do it contains code already to add the correct Bearer headers in for Oauth if you use  the OAuthCredentials class


        $OAuthCredentials = New-Object Microsoft.Exchange.WebServices.Data.OAuthCredentials((ConvertFrom-SecureStringCustom -SecureToken $Script:Token.access_token))
        $service.Credentials = $OAuthCredentials

If your using the ADAL library be aware while its correct to say it does have a TokenCache and code to refresh the tokens once they expire this won't work with the EWS Managed API. As you can see from the code above because you only pass in the Access_Token (as a String) into this class it doesn't do any active management of the Token from that point. This means if you just pass in the AccessToken from whatever method you used to generated it if your script runs for more then one hour your code will fail at the 60 minute point. Unfortunately the EWS Managed API doesn't have a CallBack to use where you could link in the ADAL libraries refresh function (or your own code to manage the token refresh) to check before making a request. So one important modification you should make to your code if before it makes an EWS call (eg Bind,Load,findItems etc) you will need to add in code to check for expired tokens. The ADAL has the acquiretokensilentasync method for this (this will return the token from cache or renew if necessary), in the code in my example I have the Invoke-ValidateToken function which does the renewal after 50 minutes to avoid token expiration.

Application registrations

Its always a good idea to create your own application registration as this gives you the ultimate control over what rights a script or application will have in your environment (although EWS really only has one grant that will work which is full Mailbox Access). But creating you own ApplicationId's allows you  track the use of the ApplicationId and potential audit the miss use of a particular application more effectively. For scripts using the Native application type and the Out of Band redirect(urn:ietf:wg:oauth:2.0:oob) is generally a good idea as you won't have an registered endpoint online to redirect the Authentication to once its complete. There are some good walk throughs to do this https://blogs.technet.microsoft.com/dawiese/2017/04/15/get-office365-usage-reports-from-the-microsoft-graph-using-windows-powershell/ shows the newer portal screens.

Last word on security

While oAuth is a great improvement in security over basic Authentication its not the panacea in itself for the InfoSec issues the IT industry faces as a whole. So you should see implementing oAuth as a step in the right direction but be careful and always treat access tokens like you would usernames and passwords (eg don't store them in plain text) and look at always strengthening your authentication with measures like Multi-Factor Authentication. 


The script includes a Test function which just binds to the Inbox so you use it like

Import-Module .\GenericOauthEWS.ps1 -Force

Test-EWSConnection -MailboxName gscales@datarumble.com

Testing

Like any script you get off the Internet you should always do your own testing but consider these points of failure for Oauth tokens that you wouldn't have to have previously considered with Basic Auth

  • If things run for more the 1 hour does my token renew correctly
  • If the AccessToken suddenly becomes Invalid for X Reason and I get a 401 error in my code will it handle the reauth (using the RefreshToken). (for the second part you need to build in an exception handler in you code for 401 error when using oAuth)
Regional Azure Endpoints

This script is hardcoded to use the common (Production) Azure Authentication endpoint that the majority of Office365 Tenants would use however some regions like China,Germany and US Government have their own dedicated endpoint so you would need to change the URL in this case see https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/extending-sharepoint-online-for-germany-china-usgovernment-environments . There are ways of discovering the Endpoint dynamically which I'll included in future updates of the script.

Thursday, July 19, 2018

Expanded Folder Age and Attachment reporting for Exchange / Office 365 Mailboxes with EWS and Powershell

Folder Item numbers, size's and Attachment Item Numbers, types and sizes are one of the Mailbox statistics that is useful when your trying to look at how and why a mailbox is growing or why your mailbox performance might be declining if you have a large number of folder items. With the recent increases in Mailbox size quotas  in Office365 it can be easy to become quite indifferent to how quickly your mailbox might be growing. For example if you have 100GB quota the lazy mindset (which I admit to being a part of) can be to just keep everything you get via mail (or stack it and rack it). If your users are not able to take advantage of cloudy attachments (or the users refuse to change their use habits) the nature of expanding file sizes means a sharper growth curve for Folder Items and Attachments.

The traditional way of tracking Mailbox sizes is to use Get-MailboxStatistics and Get-MailboxFolderStatistics for which there are many good scripts around that give a good overview of what is in a Mailbox or Folder. For a deeper dive into what  is in messages in a particular folder where you are seeing growth then using a Mailbox API like EWS (or REST) can be useful. In the past I've posted a few scripts for doing Attachment and Item level statistics which where in need of some updating which is what I've done with this post

Script Approach 

For this script I've used EWS as it is universal across OnPremise and Office365 from Exchange 2010 up. I do have two different versions of this script however because Office365 and Exchange 2016 allows you to have reference attachments (cloudy attachments) so some slightly different code is needed and the latest compiled version of the EWS Managed API which is only available from GitHub is needed so statistics can be compiled successfully for these type of Attachments. For the Item level processing side of the script it will process every item in a folder firstly using the EWS FindItems operation which will return a minimal results set of Items and Sizes and then for those items with Attachments another batch operation is used to get the metadata for the attachments which will include the Attachment name (which we can then derive the attachment type from using the file extension name) and the size of the attachment. If its a cloudy attachment then this is processed separately. I haven't included any code to call out to OneDrive to get the filesize for the cloudy attachments but its possible to do this with some extra code (generally you would want to do this asynchronously while processing the Exchange attachments).

The Results the script produces is the following html report



The script writes its result sets out in JSON, I've then used the Tabulator JavaScript library http://tabulator.info/ which can turn JSON datasets into well formatted tables and offer sorting options so its quite easy to create a more user friendly results set then just a plain CSV file which I usually use. Currently the script will report on just one particular folder as I envisaged that it would be best used in conjunction with other scripts (that utilize Get-MailboxStatistics and Get-MailboxFolderStatistics eg take your pick from https://social.technet.microsoft.com/Search/en-US/office?query=Mailbox%20size%20report ) where you identify a particular Mailbox or  folder you want to investigate more thoroughly as this script can take some time to run. I also have two defined modes where you can process Inline Attachments (which are primary just image files) or skip the processing of Inline attachments which will make the script run a lot faster because of the number of Messages that have inline attachments.

Some Examples of running the script on the Inbox folder without processing Inline attachments


Import-Module .\AttachmentStats.ps1
Invoke-MailboxFolderAttachmentReport -MailboxName gscales@dataru mble.com -FolderPath \Inbox -Verbose

Or with processing Inline attachments

Import-Module .\AttachmentStats.ps1
Invoke-MailboxFolderAttachmentReport -MailboxName gscales@dataru mble.com -FolderPath \Inbox -IncludeInline -Verbose
The report will be saved to the same path as the script module is located.

The regular version of the script can be found on GitHub https://github.com/gscales/Powershell-Scripts/blob/master/AttachmentStats.ps1

The cloudy version of the script can be found on GitHub here https://github.com/gscales/Powershell-Scripts/blob/master/AttachmentStats-Cloudy.ps1 

The cloudy version will require the following EWS Managed API version https://github.com/gscales/Powershell-Scripts/raw/master/Microsoft.Exchange.WebServices.dll to be located in the same directory as the script module.

The Cloudy version will produce a separate table of cloudy attachment stats eg



Hire me - If you would like to do something similar to this or anything else you see on my blog I'm currently available to help with any Office365, Exchange or Active Directory related development work or scripting, please contact me at gscales@msgdevelop.com(nothing too big or small).