Modifying your EWS WSDL Proxy Code for Modern Authentication

This is a follow-on from my last post on Modifying your EWS Managed API code to use Hybrid Modern Authentication against OnPrem Mailboxes . If instead of the EWS Managed API you are using EWS Proxy Code (generated from the EWS WSDL) and you want to migrate it to using Modern Authentication for Office365 and/or Hybrid here's a method you can use using the MSAL Authentication library.

Unlike the EWS Managed API the WSDL generated proxy classes and specifically the ExchangeServiceBinding class doesn't have any provision to use Token Credentials. One way of implementing this in .NET is to take advantage of  Polymorphism and create a new class that is derived from the ExchangeServiceBinding class and then override the method GetWebResponse from this class (which is actually derived from the SoapHttpClientProtocol class which contains the actual method we are going to override https://docs.microsoft.com/en-us/dotnet/api/system.web.services.protocols.soaphttpclientprotocol.getwebrequest?view=netframework-4.8 )

At the same time we can also add the X-AnchorMailbox header into the request which is also recommended for any Exchange Online requests you make. And because this method is called before every EWS Request we can place our Token Refresh code in there. In this example I'm using which uses the MSAL all you need to include is code that fetches the token from the TokenCache, this will trigger a Token Refresh if need or ultimately throw to Interaction if the Refresh Token isn't available. So here is a basic C# Console App that can do Hybrid/Modern Auth discover using the MSAL library. If you want the project files you can download them from here

 

Modifying your EWS Managed API code to use Hybrid Modern Authentication against OnPrem Mailboxes

In this post I'm going to look at what you need to do in your EWS Managed API code to support using Hybrid Modern Authentication where previously you've been using Basic or Integrated Authentication (both of which are susceptible to password spray attacks). If you don't know what Hybrid Modern Authentication  is put simply it brings to Exchange OnPrem email clients the security benefits of Modern Authentication offered by Azure AD to Office365 tenants. If your already using OAuth to connect to Office365 you have most of the work already done but you will still need logic to ensure you have the correct Audience set in your token when that code is used against an OnPrem Mailbox. 

Prerequisites 

You need to be using Hybrid Exchange or more specifically 

Hybrid Office 365 tenant is configured in full hybrid configuration using Exchange Classic Hybrid Topology mode ref https://docs.microsoft.com/en-us/exchange/clients/outlook-for-ios-and-android/use-hybrid-modern-auth?view=exchserver-2019 

And you need to configure Hybrid Modern Authentication https://docs.microsoft.com/en-us/office365/enterprise/configure-exchange-server-for-hybrid-modern-authentication 

If you don't want to enable Hybrid Modern Authentication but still want to use oAuth in EWS you can do it and there is a good article by Ingo on how to do this  https://practical365.com/exchange-server/configure-hybrid-modern-authentication-for-exchange-server/

Authentication - Acquiring the Token

This is where you need to make the most changes in your current code as you will now need some logic that can be used to acquire the oAuth Tokens from AzureAD. The easiest way of doing this is to use one of the Authentication libraries from Microsoft either ADAL (if you already have this implemented in your code) or preferably use the MSAL library. The difference between ADAL and MSAL is ADAL uses the v1 Azure oauth endpoint and MSAL uses the v2 there is a good description of the differences between the two endpoints  https://nicolgit.github.io/AzureAD-Endopoint-V1-vs-V2-comparison/

Getting the intended Audience value for you Token Request 

The audience of an oAuth token is the intended recipient of the token (or basically the resource its going to be used against) , in our Exchange EWS context this is the host-name part of the EWS External endpoint. In Office365 the EWS Endpoint will be https://outlook.office365.com/ews/exchange.asmx so the intended Audience of a token will be https://outlook.office365.com if you look at an Office365 token is jwt.io this is what you see eg

When your using Hybrid Modern Authentication the Audience value for your token will become the external EWS endpoint's host-name of your OnPrem server (generally what you have configured in get-webservicesvirtualdirectory)

In the Authentication libraries this Audience is passed differently in

ADAL v1 Azure Endpoint its passed as the resourceURL

String ResourceURL = ""https://outlook.office365.com";
var AuthResults = AuthContext.AcquireTokenAsync(ResourceURL ,
 "xxxxx-52b3-4102-aeff-aad2292ab01c", new Uri("urn:ietf:wg:oauth:2.0:oob"),
 new PlatformParameters(PromptBehavior.Always)).Result; 

In MASL v2 Azure Endpoint its passed as part of the scope 

string scope = "https://outlook.office365.com/EWS.AccessAsUser.All";
PublicClientApplicationBuilder pcaConfig =
 PublicClientApplicationBuilder.Create(ClientId).WithAuthority(AadAuthorityAudience.AzureAdMultipleOrgs);
var IntToken = pcaConfig.Build().AcquireTokenInteractive(
new[] { scope }).ExecuteAsync().Result;

With Hybrid Modern Auth in the above examples outlook.office365.com would be replaced with the host name for your external EWS endpoint which you would obtain usually via Autodiscover

AutoDiscover 

Autodiscover in Exchange from Exchange 2007 has been there to help you basically discover the internal or external endpoint you need for whatever API your using. It is however an Authenticated Endpoint so when you remove Basic/Intergrated Authentication your code needs to be able to deal with this change. There are two ways you could go about this the first is generate a OAuth token first and use that to make the Authenticated traditional Auto-discover request. Or the second way is to use Autodiscover v2 (or Autodiscover json) which allows you to make an unauthenticated autodiscover requests to return the API endpoint you want. 

If your code targets predominately Office365 and Hybrid tenants then just switch to use Autodiscover v2, if you have a mix of Office365, Hybrid and OnPrem islands then you still need the legacy Basic/Integrated Auth method for these OnPrem clients. The Approach that I'm taking in this post is to first do a Realm discovery against Office365 to determine if a particular set of credentials is an Office365 or Hybrid account.If it is then a v2 Autodiscover request will be made against Office365  and if not fail back to the legacy code. This isn't 100% guaranteed to work for some OnPrem (especially pre Exchange 2016) and account combinations so my advice is you always make sure your try/catch autodiscover logic includes at least one legacy auto discover-attempt as a last fail back. And make sure you do some regression testing on your code change against Exchange 2013. 

For doing a simple JSON based Autodiscover against Office365 this can be done in a few lines with httpclient in c# 

String MailboxName = "gscales@datarumble.com";
String EWSEndPoint = $"https://outlook.office365.com/autodiscover/autodiscover.json/v1.0/{MailboxName}?Protocol=EWS";
HttpClient httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (compatible; AcmeInc/1.0)");
dynamic JsonResult = JsonConvert.DeserializeObject(httpClient.GetAsync(EWSEndPoint).Result.Content.ReadAsStringAsync()
.Result);
Console.WriteLine(JsonResult.Url);

Or in PowerShell you could do it as a one-liner

(Invoke-WebRequest -Uri https://outlook.office365.com/autodiscover/autodiscover.json/v1.0/gscales@datarumble.com
?Protocol=EWS | ConvertFrom-Json).url

When your submitting an Autodiscover request against Office365 if your mailbox is OnPrem and you have HMA configured you will get returned your OnPrem EWS endpoint. 

EWS Managed API 

So what does this look like in the context of your EWS Managed API code, let first look at the traditional code path for Autodiscover

ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2013);
service.Credentials = new WebCredentials("user1@contoso.com", "password");
service.AutodiscoverUrl("user1@contoso.com", RedirectionUrlValidationCallback);

Here is what it would look like use a Realm Discovery and then a Json Autodiscover and some MSAL code to do the Token Acquisition otherwise following the above logic.

A PowerShell version of the same thing would look like

If you want a library free version that uses Invoke-WebRequest it would look like

Dealing with Token Refresh (Important)

Access tokens by default in Azure are valid for 1 hour, so if your application is going to run for a long period of time or is persistent then you will need to manage token expiration and refresh. If your using one of the Authentication libraries then they can perform this for you automatically however they do rely on you calling their methods before you make any authenticated EWS call. Currently the EWS Managed API doesn't offer a callback to help you integrate easily with an Authentication library (for doing the Token Refresh management) so you will need to come up with your own method of doing this (eg a simple method to check the token before any operation could be used). Or you can modify the EWS Managed API source to integrate your own callback eg a good place to look is PrepareWebRequest in https://github.com/OfficeDev/ews-managed-api/blob/70bde052e5f84b6fee3a678d2db5335dc2d72fc3/Credentials/OAuthCredentials.cs . The good thing about modifying the source is that you fix the issue for any operation that you code will do now and into the future. 

Graph Mailbox Basics with PowerShell Part 1 Folders

I haven't done a basics series for a while but based on some of the questions I've been getting lately and the lack of some good Mailbox specific examples for basic but more complex tasks using the Graph against Exchange Online Mailboxes this seemed like a good series to write.

For all the scripts in this series I'm not going to use any modules or other libraries so everything will be using Invoke-WebRequest and Invoke-RestMethod, while there is nothing wrong with using libraries or modules and a number of advantages in doing so it just keeps the examples as simple and easy to understand as they can be.

Authentication You can't have an article on the Graph without talking about authentication and we are now far from the past where all you needed was a simple username and password and you where off to the races. The basics of Authentication are is that first you will need an Azure App Registration (that has been consented to), there are many pages dedicated to how you can do this  (this is one of the better ones) so I'm not going to dwell too much on this. My simple template script has a function called Get-AccessTokenForGraph which takes a ClientId and RedirectURI and does an interactive login to get the Azure access token. With oAuth there are many other ways of authenticating so if this doesn't fit your needs you just need to plug your own code in the Get-AccessTokenForGraph function.

Get-FolderFromPath

With Exchange the locator (think file path as an analogy) you use to access a Folder programatically is its FolderId. Every Exchange API has it own interpretation of the FolderId starting with the Fid and PidTagEntryId in Mapi, EWS has the EWSid and Graph just has the Id (and the EMS gives a combination of Id's back depending on which cmdlet you use). With the Graph and EWS id's these id's contain the PidTagEntryId with a bunch of other flags that tell the service how to locate and open the folder. However most of the time us humans think of folders in terms of Paths eg if I have a Subfolder of the Inbox a more human reference would be \Inbox\subfolder (language differences aside). So one of the more common methods I use is the Get-FolderFromPath to get a folder (or just the folderid) so you can then work on the Items within that folder or the folder itself. So the method I've always used in EWS is to take the path you want to search for and split in based on the \ character and then do a number of shallow searches of the parent folders until you find the child folder you want. in the Graph this looks something like this

        $RequestURL = $EndPoint + "('$MailboxName')/MailFolders('MsgFolderRoot')/childfolders?"
        $fldArray = $FolderPath.Split("\")
        $PropList = @()
        $FolderSizeProp = Get-TaggedProperty -DataType "Long" -Id "0x66b3"
        $EntryId = Get-TaggedProperty -DataType "Binary" -Id "0xfff"
        $PropList += $FolderSizeProp 
        $PropList += $EntryId
        $Props = Get-ExtendedPropList -PropertyList $PropList 
        $RequestURL += "`$expand=SingleValueExtendedProperties(`$filter=" + $Props + ")"
        #Loop through the Split Array and do a Search for each level of folder 
        for ($lint = 1; $lint -lt $fldArray.Length; $lint++) {
            #Perform search based on the displayname of each folder level
            $FolderName = $fldArray[$lint];
            $headers = @{
                'Authorization' = "Bearer $AccessToken"
                'AnchorMailbox' = "$MailboxName"
            }
            $RequestURL = $RequestURL += "`&`$filter=DisplayName eq '$FolderName'"
            $tfTargetFolder = (Invoke-RestMethod -Method Get -Uri $RequestURL -UserAgent "GraphBasicsPs101" -Headers $headers).value  
            if ($tfTargetFolder.displayname -match $FolderName) {
                $folderId = $tfTargetFolder.Id.ToString()
                $RequestURL = $EndPoint + "('$MailboxName')/MailFolders('$folderId')/childfolders?"
                $RequestURL += "`$expand=SingleValueExtendedProperties(`$filter=" + $Props + ")"
            }
            else {
                throw ("Folder Not found")
            }
        }

So for each folder Step I'm finding the intermediate folder using $filter=DisplayName eq '$FolderName'

To make the results more useful I've included a few extended properties that give me some extra information

The first is the FolderSize, which in Mapi is the PidTagMessageSizeExtended property on the folder 

The second is the pidTagEntryId (PR_EntryId)property which I added in so I could easily convert this into the folderId format that is used in the Office365 compliance search eg in Office365 when you do a compliance search you have the ability of using the folderid:xxxx keyword in a Search to limit the search of a Mailbox to one particular folder in a Mailbox. There is a script in https://docs.microsoft.com/en-us/microsoft-365/compliance/use-content-search-for-targeted-collections?view=o365-worldwide which uses the Get-MailboxFolderStatistics cmdlet which I found a little cumbersome so having a simple method like the above can return the id i need for the folder i want. eg this is what the end result looks like when you run the script

The REST request that is generated by the script looks like (if you want to try this in the graph explorer)

https://graph.microsoft.com/v1.0/users('gscales@datarumble.com')
/MailFolders('MsgFolderRoot')/childfolders?
$expand=SingleValueExtendedProperties($filter=(Id%20eq%20'Long%200x66b3')
%20or%20(Id%20eq%20'Binary%200xfff'))
&$filter=DisplayName%20eq%20'inbox'

There are a bunch more things you can do with this type of query eg working with the retention tags on a folder. Or using the FolderId to then process the Items within that folder. The reason i started with this function is for me its always a jumping off point for starting working with mailbox data.

The full script is available on GitHub  https://github.com/gscales/Powershell-Scripts/blob/master/Graph101/FolderOps.ps1

Migrating your Mailbox searches in EWS to the Graph API Part 2 KQL and new search endpoints

This is part 2 of my blog post on migrating EWS Search to the Graph API, in this part I'm going to be looking at using KQL Searches and using the new Microsoft Search API (currently in Beta). The big advantage these type of searches have over using SearchFilters is that these type of searches use the content indexes which can improve the performance of searches when folder item counts get high. They also allow you to query the contents of  Attachments which are indexed through ifilters on the server.

KQL queries on the Mailbox and Mailbox Folders

In EWS you have been able to use firstly AQS and now KQL in the FindItems operation from Exchange 2013 up. To migrate these searches to Microsoft Graph is pretty simple eg an EWS FindItem query to search for all messages with a pdf attachment

FindItemsResults fiItems = service.FindItems(QueryFolder, "Attachmentnames:.pdf", iv);

in the Graph you would use something like

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages
?$search="attachmentnames:.pdf"

the slightly disappointing thing with the Graph is that you can't use count along with a search which when your doing statistical type queries eg say I wanted to know how many email that where received in 2019 had a pdf attachment makes this very painful to do in the Graph where in EWS it can be done with one call (its a real snowball that one).

Searching the recipient fields like To and CC, in the forums you see some absolute clangers search filters that try to search the recipients and from fields of messages that can easily be done using the participants keyword which includes all the people fields in an email message. These fields are From, To, Cc. The one thing to be aware of is the following note on expansion in https://docs.microsoft.com/en-us/microsoft-365/compliance/keyword-queries-and-search-conditions?view=o365-worldwide . So if you don't want expansion to happen you need to ensure you use the wildcard character after the participant your searching for. A simple participants query looks like

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$search="participants:Fred"

Date range queries

One of the good things about KQL with dates is that you can use reserved keywords like today,yesterday,this week eg

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')
/messages?$search="received:yesterday"

to get all the received sent between two dates you can use either

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$search="received:2019-01-01...2019-02-01"

or

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$search="(received>=2019-01-01 AND received<=2019-02-01)"

If you want to search the whole of the Mailbox using the graph eg if you have use the AllItems Search Folder in EWS to do a Search that spans all the MailFolders in a Mailbox in the Graph you just need to use the /Messages endpoint eg

https://graph.microsoft.com/v1.0/me/messages?
$search="(received>=2019-01-01 AND received<=2019-02-01)"

New Search Methods

The traditional search methods in EWS give you the normal narrow refiner search outputs that most mail apps have been providing over the past 10-20 years. While these methods have improved over the years there hasn't been any real great leaps in functionality with Search. So the Microsoft Graph has been adding some newer endpoints that do allow a more modern approach to searching . The first is Microsoft Graph data connect which has been around for a while now and the Microsoft Search API which is still in Beta. As this article is about migrating EWS searches you probably wouldn't consider either of these for your traditional search migration as $filter and $search are going to meet those needs. However if you are looking at overhauling the search functionality in your application or you are building greenfield functionality then both of these new methods are worth consideration.

Microsoft Graph Data Connect

https://docs.microsoft.com/en-us/graph/data-connect-concept-overview

Graph Data connect is your go-to endpoint when you want to do any mass processing of Mailbox data. It solves that problem of having to crawl every item in a Mailbox when you want to do any data-mining type operations by basically providing an Azure dataset of this information for you. Data connect is great however it has a high entry level, first you need a Workplace analytics licence for every mailbox you wish to analyse and the costs can mount pretty quickly the larger the Mailbox count your dealing with. The other requirements is paying for the underlying Azure Storage etc that your dataset ends up consuming. I think it can be a bit of a shame that the licencing costs can lock a lot of  developers out of using this feature as it really does provide a great way or working with Mail item data. And it leaves some having to resort to doing their own crawling of Mailbox data to avoid these costs (eg that licencing cost is a pretty hard sell for any startup looking to use this) 

Microsoft Search API

https://docs.microsoft.com/en-us/graph/search-concept-overview

This is the newest way of searching mailbox data, while the underlying mechanism for doing mailbox searches is still KQL so its very similar to the $Search method described about,  this API does enhance the search results with some more "Search Intelligence" like relevance bringing AI into the picture . One of the other main benefits of this endpoint is when you want to broaden your search to other Office365 workflows or even include your own custom data searches. So this really is the endpoint that will provide you with a modern search experience/workflow. Which is getting more critical due to the sheer amount of data we have (eg the datageddon). Its still in beta and is a little restricted at the moment eg

• It can't be used to search delegate Mailboxes so only the primary mailbox 
• It only returns the pageCount for items not the Total number of Items found in a search (to be fair $search does this as well which is really annoying)
• Searches are scoped across the entire mailbox 
• Just Messages and Events are searchable at the moment

Migrating your Mailbox searches in EWS to the Graph API Part 1 Filters and Search Folders

This is part one of a two part post where I'm going to look at how you can migrate any searches you are doing in EWS to the Graph API. In this first part I'm going to cover SearchFilters (from EWS) and Search-Folders as they have been around the longest and in part 2 I'll look at Searches which has  some new functionality in beta in the Graph.

Lets start by looking at how you might be doing searches in EWS at the moment

• Search Filters (restrictions) in a FindItem Request that can be run against a Folder or Search Folder
• QueryString (KQL) in a FindItem Request that can be run against a Folder or Search Folder
• SearchFolder with a FindItem Request
• eDiscovery via SearchMailbox which has now been depreciated in Office 365 and no longer supported

Search Filters (Restrictions)

If you have used the EWS Managed API to build your application you use the SearchFilter class which creates a underlying restriction in EWS https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/restriction.  The term Restriction came from the Exchange ROP's protocol which is what MAPI uses to talk to the Exchange Store.

In the Microsoft Graph the language you talk in regards to filtering is OData  

OData (Open Data Protocol) is an ISO/IEC approved, OASIS standard that defines a set of best practices for building and consuming RESTful APIs ref https://www.odata.org/

OData filters are therefore a standard that anybody implementing the protocol (like Microsoft have done with the Graph API) should adhere to.

Lets look at some examples (from this blog) of SearchFilters I've used and how they can be converted to Graph oData Filters.

Easy - the easiest query to make is against one of the strongly typed properties like Subject or Sender eg in EWS you might have a search filter like this

SearchFilter SubjectFilter = new SearchFilter.IsEqualTo(ItemSchema.Subject, "Subject");

in the Graph this would just look something like this (applied just against the Inbox)

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?$filter=subject eq 'test'

This is an Equality search some other things you can do is

Startswith (which you couldn't actually do in EWS)

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?$filter=startswith(Subject,'test')

Sub-String Searches

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?$filter=Contains(Subject,'test')

With the later two searches if you have a folder with a large item count then these aren't going to perform like an equality or a Content Index Search would and its a possibility that they will timeout the first time you run the query. (Then on subsequent queries may succeed this is due to the way Exchange applies temporary restrictions for dynamic searches).

Medium - The most used search filter in EWS for me is that date restriction where you want to restrict the emails returned to a certain time frame so an EWS Search Filter like the following

$Sfgt = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsGreaterThan
([Microsoft.Exchange.WebServices.Data.ItemSchema]::DateTimeReceived, $Startdatetime)
$Sflt = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsLessThan
([Microsoft.Exchange.WebServices.Data.ItemSchema]::DateTimeReceived, $Enddatetime)
$sfCollection = new-object Microsoft.Exchange.WebServices.Data.SearchFilter
+SearchFilterCollection([Microsoft.Exchange.WebServices.Data.LogicalOperator]::And);
$sfCollection.add($Sfgt)
$sfCollection.add($Sflt)

In the Graph this would look like

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$filter=(receivedDateTime gt 2020-03-24T13:01:50Z) AND (receivedDateTime lt 2020-03-25T12:59:50Z)

(Just watch the date formatting)

Hard - If your using Extended properties in your SearchFilters then you need to include the Extended property definition in the Filter and use the lambda any or all expression

The first Query looks for Items based on the underlying Message Class (which isn't exposed as a strongly typed property in the Graph)

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$filter=singleValueExtendedProperties/any(ep:ep/id eq 'String 0x001a' and ep/value eq 'IPM.Note')

Another useful filter is to be able to find Messages where a particular property has been set eg this filter looks for messages that have the In-Reply-To header set (So Repsonses only)

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$filter=singleValueExtendedProperties/any(ep: ep/id eq 'String 0x1042' and ep/value ne null)

For non String properties for instance the Message size you need to make sure you cast the value to the Json datatype. Eg to find all messages that are larger the 10 MB you could use

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$filter=singleValueExtendedProperties/any(ep:ep/id eq 'Integer 0x0E08' and cast(ep/value, Edm.Int32) gt 1048576)

In EWS when you did a search that returns a large number of Items that was paged you got back also the total number of items that matched your search query. I've used this in the past to get statistical information about email with a filter without needing to page all the results. The Graph offers the same thing using the $count query parameter eg if I change the above query to find all messages above 1 MB in my mailbox and include the $count parameter this is what I get for my mailbox

Another example of this maybe to look at the count of messages in 2018

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$count=true&$filter=(receivedDateTime gt 2018-01-01) AND (receivedDateTime lt 2019-01-01)

And finally one last example is for pidTagHasAttachments because I know myself and others use this (because it gives a different value than the strongly type hasAttachments for various reasons)

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$filter=singleValueExtendedProperties/any(ep:ep/id eq 'Boolean 0x0E1B' and cast(ep/value, Edm.Boolean) eq true)

Hopefully I've covered off enough examples here for anybody stuck with syntax to be able to get their head around it. If you do have problems try posting a question into stack overflow here

SearchFolders

SearchFolders give you a way of creating a Virtual Folder that represent an underlying restriction (or Search) that can span one folder or the whole mailbox. While they are more suited to static type searches if you have ever used the mapi fiddler inspector https://github.com/OfficeDev/Office-Inspectors-for-Fiddler to look at what Outlook is doing under the covers when you do a Search, you can see that Outlook uses Searchfolders to provide a more functional search for dynamic queries.

Another example that is used in the Microsoft graph is the me/Messages endpoint which is a Searchfolder that provides access to all the mail folders in a mailbox.

In EWS when you create a SearchFolder you specify a SearchFilter for that folder to be based on. With the Graph similarly you can create a SearchFolder based on a OData filter which I've detailed above. So looking at something topically if you wanted to create a SearchFolder to show all the Email which had a subject of Coronavirus you could use

{
  "@odata.type": "microsoft.graph.mailSearchFolder",
  "displayName": "Coronavirus Email",
  "includeNestedFolders": true,
  "sourceFolderIds": ["AQMkADYA…."],
  "filterQuery": "contains(subject, 'Coronavirus')"
}

One thing that is mentioned in the SearchFolder documentation https://docs.microsoft.com/en-us/graph/api/resources/mailsearchfolder?view=graph-rest-1.0 for the Graph to be aware of is

1. Search folders expire after 45 days of no usage.
2. There are limits on the number of search folders that can be created per source folder. When this limit is breached, older search folders are deleted to make way for new ones.

So if you are going to use SearchFolders in your application you will need to make sure you have some appropriate management logic. Searchfolders are pretty powerful like EWS the Graph only implements a subset of what can be done in MAPI so if you are trying to reproduce what you see is possible in Outlook you may not be able to do this with Graph (or EWS).

Automating opening a Search-Mailbox result in Excel using EWS

While the Search-Mailbox cmdlet is now depreciated in Exchange Online, OnPrem its still used a fair bit and also does still have some use in the cloud for specific tasks. I've been using it this week a fair bit for various testing tasks and one pain I found when doing a lot of repeated searches in logging mode is each time to have to go in, open the results message in the discovery search mailbox and download the attachment with the log file, unzip and open it in Excel. So I came up with a way of automating this in powershell which turned out to be pretty simple but effective.

First off the only information you need to get the Results Message gets returned in the Target Folder property of the Search results eg.

 The TargetFolder value tells you what folder in the discovery Search mailbox the results are stored in and the DateTime value that will be in the subject of the Results Message.

So in EWS you can use FindFolder to Find that Folder (using a Split on "\" which will work as long as you don't put that in the displayName) and then FindItem can be used to find the results Item eg.

        $ivItemView = New-Object Microsoft.Exchange.WebServices.Data.ItemView(1)  
        $SfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+ContainsSubstring([Microsoft.Exchange.WebServices.Data.ItemSchema]::Subject, $Subject) 
        $findItemResults = $Folder.FindItems($SfSearchFilter, $ivItemView)
        if ($findItemResults.Items.Count -eq 1) {
            return $findItemResults.Items[0]
        }
        else {
            throw "No Item found"
        }

Once you have the Results Message you can download the Attachment using some code like this

        if ($SearchResultItem.HasAttachments) {
            $SearchResultItem.Load();
            foreach ($Attachment in $SearchResultItem.Attachments) {
                $Attachment.Load()

I then save it to the default downloads directory using

                $downloadDirectory = (New-Object -ComObject Shell.Application).NameSpace('shell:Downloads').Self.Path
                $fileName = ($downloadDirectory + "\" + $ItemPath.SubString(1).Replace("/", "-").Replace(":", "-") + "-" + $Attachment.Name.ToString())
                $fiFile = new-object System.IO.FileStream($fileName, [System.IO.FileMode]::Create)
                $fiFile.Write($Attachment.Content, 0, $Attachment.Content.Length)
                $fiFile.Close()

and finally open the ZipFile, Extract the csv datastream from the Archive and save that as a File in the Downloads Directory and then open that file in Excel.

                if ($FileName.contains(".zip")) {                    
                    $Zip = [System.IO.Compression.ZipFile]::OpenRead($FileName)
                    try {
                        foreach ($file in $Zip.Entries) {
                            if ($file.Name.contains(".csv")) {
                                $ms = New-Object System.IO.MemoryStream
                                $ZipStream = $file.Open()
                                $ZipStream.CopyTo($ms);
                                $outputfile = $FileName.replace("zip", "")
                                [System.IO.File]::WriteAllBytes($outputfile, $ms.ToArray())
                                Invoke-Item $outputfile
                                
                            }
                        }
                    }
                    catch {
                        Write-Host $_.ScriptStackTrace
                    }
                    $Zip.Dispose()

An example of using this is

$SearchResult = Search-Mailbox -id meganb -TargetFolder Search1 -SearchQuery from:glen -TargetMailbox "DiscoverySearchMailbox{D919BA05-46A6-415f-80AD-7E09334BB852}@xxx.onmicrosoft.com" -LogOnly -LogLevel Full

then

Get-SearchMailboxResultsToExcel -MailboxName "DiscoverySearchMailbox{D919BA05-46A6-415f-80AD-7E09334BB852}@M365x680608.onmicrosoft.com" -SearchResultPath $SearchResult.TargetFolder -Verbose

or if you want to use ModerAuth (You will need the Adal dll in the same directory)

Get-SearchMailboxResultsToExcel -MailboxName "DiscoverySearchMailbox{D919BA05-46A6-415f-80AD-7E09334BB852}@M365x680608.onmicrosoft.com" -ModernAuth -SearchResultPath $SearchResult.TargetFolder -Verbose

I've put a download of this script on GitHub https://github.com/gscales/Powershell-Scripts/blob/master/Get-SearchMailboxResultsToExcel.ps1

Export calendar Items to a CSV file using Microsoft Graph and Powershell

For the last couple of years the most constantly popular post by number of views on this blog has been Export calendar Items to a CSV file using EWS and Powershell closely followed by the contact exports scripts. It goes to show this is just a perennial issue that exists around Mail servers, I think the first VBS script I wrote to do this type of thing was late 90's against Exchange 5.5 using cdo 1.2.

Now it's 2020 and if your running Office365 you should really be using the Microsoft Graph API to do this. So what I've done is create a PowerShell Module (and I made it a one file script for those that are more comfortable with that format) that's a port of the EWS script above that is so popular. This script uses the ADAL library for Modern Authentication (which if you grab the library from the PowerShell gallery will come down with the module). Most EWS properties map one to one with the Graph and the Graph actually provides better information on recurrences then EWS did. Where extended properties where used in the EWS script the equivalent is used in the Graph. (The only real difference is the AppointmentState property which is a strongly typed property in EWS but I had to use the Extended property in the Graph).

Just a couple of things if your new to Microsoft Graph scripts and Modern Authentication that you need to know

1. You need an Approved Azure Application registration to use this (or any script that is going to access the Graph). The Microsoft walk-throughs https://docs.microsoft.com/en-us/graph/auth-register-app-v2 are pretty good at describing how to do this. Specific config I recommend you use

"https://login.microsoftonline.com/common/oauth2/nativeclient" as the redirectURL (this is part of the Suggested Redirect URIs for public clients (mobile, desktop)).

2. Permission for the above

You only need the following permissions for this script to work, Calendar.Read gives you rights to the calendar the account that is being used and Calendar.Read.Shared gives you read access to any calendars that the account being used has been granted access to (eg via delegation, admin portal or add-mailboxpermission). 

Then you just need to copy the Application (client) ID guid from the overview screen in the Applicaiton Registration  and use that as in the -clientId paraemter in the Export-GCECalendarToCSV cmdlet.

I've included a demo multi tenant app registration as the default in the module that just has these rights which you can use for testing but I would always recommend you create you own.

You can install the module which will give you access to the Export-GCECalendarToCSV and Export-GCECalendar cmdlets from the Powershell gallery https://www.powershellgallery.com/packages/MSGraph-ExportCalendar/  (see the instruction on that page).

Or if you want to take the script and modify it yourself its located on GitHub https://github.com/gscales/Powershell-Scripts/blob/master/MSGraph-ExportCalendar/functions/Export-GCECalendarToCSV.ps1

Simple example of exporting the last 7 days of calendar appointment to csv

Export-GCECalendarToCSV -MailboxName gscales@datarumble.com -StartTime (Get-Date).AddDays(-7) -EndTime (Get-Date) -FileName c:\temp\Last7.csv

Using Azure device code authentication on a arduino iot 33 and getting the Teams presence from the Microsoft Graph

A while ago I published this post on accessing the Graph directly from an Arduino, this made use of the "resource owner password credentials grant" (meaning it used a hard coded username and password). Once you have enabled MFA (multi factor authentication) on an account this grant no longer works because you have no ability to provide the other factors for the Authentication to succeed.  For devices like Arduino's or most IOT devices that have very limited UI capabilities this is where device code authentication can be used.

The way Device Code Authentication works is instead of posting the user credentials to the token endpoint to get an access token, you make a post first to the /v2.0/devicecode endpoint which will then give you a specific user code to use to authenticate with on another device. You then visit http://microsoft.com/devicelogin (on a pc or mobile device) enter the user code and authenticate as the required user doing any extra MFA authentication. In the meantime the limited UI device polls the Token Endpoint and once authentication has been completed(on the external device) instead of the endpoint returning a pending error the poll results will be a normal Access token (and refresh token) that can then be used to access any Graph resources you have access to.

Visually on the Serial port here is what the whole process looks like on the Arduino

The last part of this code makes a request to get the Presence from Microsoft Teams which was introduced into beta in the Microsoft Graph in December see https://docs.microsoft.com/en-us/graph/api/resources/presence?view=graph-rest-beta.

So putting this all together you can make a simple Teams presence light with a circuit like (circuit is for demonstration purposes only)

and processing the Presence result you can get returned from the Graph using the code I've referenced below

A few notes on Device code Authentication, its important when you setup your App Registration in the Azure Portal that you mark your registration as public "Treat application as a public client" eg

Device code requests must be made against the Tenant endpoint (so you can't use the common endpoint). In the code I've included discovery code that gets the tenant specific endpoint to use based on the domain name stored in the Secrets file.

Also if your reading this because your following the documentation for Device code on https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-device-code and you can't get it to work there is an issue with the payload information in the document. Where device_code is used as a parameter name in the payload in the documentation it should just be code with your device code as the value.

I've put the sketch which contains the code I've used for Device Code authentication and grabbing the presence from the Microsoft Graph on GitHub here https://github.com/gscales/MS-Graph-Arduino/tree/master/MSGraph-Presence please refer to my previous article on details on getting you code up and running on an Arduino Iot33 which include downloading the SSL certs to the device which is required (also flash the firmware).

A couple of notes on the code because the Json parsing library I used can't handle the access token response I needed to manually parse the token out (which is a little frustrating) but is one of the chanllendges of working with Arduino's and dealing with the issues that limited memory causes.