Skip to main content

Migrating your Mailbox searches in EWS to the Graph API Part 2 KQL and new search endpoints

This is part 2 of my blog post on migrating EWS Search to the Graph API, in this part I'm going to be looking at using KQL Searches and using the new Microsoft Search API (currently in Beta). The big advantage these type of searches have over using SearchFilters is that these type of searches use the content indexes which can improve the performance of searches when folder item counts get high. They also allow you to query the contents of  Attachments which are indexed through ifilters on the server.

KQL queries on the Mailbox and Mailbox Folders

In EWS you have been able to use firstly AQS and now KQL in the FindItems operation from Exchange 2013 up. To migrate these searches to Microsoft Graph is pretty simple eg an EWS FindItem query to search for all messages with a pdf attachment

FindItemsResults fiItems = service.FindItems(QueryFolder, "Attachmentnames:.pdf", iv);

in the Graph you would use something like

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages
?$search="attachmentnames:.pdf"

the slightly disappointing thing with the Graph is that you can't use count along with a search which when your doing statistical type queries eg say I wanted to know how many email that where received in 2019 had a pdf attachment makes this very painful to do in the Graph where in EWS it can be done with one call (its a real snowball that one).

Searching the recipient fields like To and CC, in the forums you see some absolute clangers search filters that try to search the recipients and from fields of messages that can easily be done using the participants keyword which includes all the people fields in an email message. These fields are From, To, Cc. The one thing to be aware of is the following note on expansion in https://docs.microsoft.com/en-us/microsoft-365/compliance/keyword-queries-and-search-conditions?view=o365-worldwide . So if you don't want expansion to happen you need to ensure you use the wildcard character after the participant your searching for. A simple participants query looks like

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$search="participants:Fred"

Date range queries

One of the good things about KQL with dates is that you can use reserved keywords like today,yesterday,this week eg

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')
/messages?$search="received:yesterday"

to get all the received sent between two dates you can use either

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$search="received:2019-01-01...2019-02-01"

or

https://graph.microsoft.com/v1.0/me/mailFolders('Inbox')/messages?
$search="(received>=2019-01-01 AND received<=2019-02-01)"

If you want to search the whole of the Mailbox using the graph eg if you have use the AllItems Search Folder in EWS to do a Search that spans all the MailFolders in a Mailbox in the Graph you just need to use the /Messages endpoint eg

https://graph.microsoft.com/v1.0/me/messages?
$search="(received>=2019-01-01 AND received<=2019-02-01)"


New Search Methods

The traditional search methods in EWS give you the normal narrow refiner search outputs that most mail apps have been providing over the past 10-20 years. While these methods have improved over the years there hasn't been any real great leaps in functionality with Search. So the Microsoft Graph has been adding some newer endpoints that do allow a more modern approach to searching . The first is Microsoft Graph data connect which has been around for a while now and the Microsoft Search API which is still in Beta. As this article is about migrating EWS searches you probably wouldn't consider either of these for your traditional search migration as $filter and $search are going to meet those needs. However if you are looking at overhauling the search functionality in your application or you are building greenfield functionality then both of these new methods are worth consideration.



Graph Data connect is your go-to endpoint when you want to do any mass processing of Mailbox data. It solves that problem of having to crawl every item in a Mailbox when you want to do any data-mining type operations by basically providing an Azure dataset of this information for you. Data connect is great however it has a high entry level, first you need a Workplace analytics licence for every mailbox you wish to analyse and the costs can mount pretty quickly the larger the Mailbox count your dealing with. The other requirements is paying for the underlying Azure Storage etc that your dataset ends up consuming. I think it can be a bit of a shame that the licencing costs can lock a lot of  developers out of using this feature as it really does provide a great way or working with Mail item data. And it leaves some having to resort to doing their own crawling of Mailbox data to avoid these costs (eg that licencing cost is a pretty hard sell for any startup looking to use this) 


Microsoft Search API

https://docs.microsoft.com/en-us/graph/search-concept-overview

This is the newest way of searching mailbox data, while the underlying mechanism for doing mailbox searches is still KQL so its very similar to the $Search method described about,  this API does enhance the search results with some more "Search Intelligence" like relevance bringing AI into the picture . One of the other main benefits of this endpoint is when you want to broaden your search to other Office365 workflows or even include your own custom data searches. So this really is the endpoint that will provide you with a modern search experience/workflow. Which is getting more critical due to the sheer amount of data we have (eg the datageddon). Its still in beta and is a little restricted at the moment eg

  • It can't be used to search delegate Mailboxes so only the primary mailbox 
  • It only returns the pageCount for items not the Total number of Items found in a search (to be fair $search does this as well which is really annoying)
  • Searches are scoped across the entire mailbox 
  • Just Messages and Events are searchable at the moment






Popular posts from this blog

Export calendar Items to a CSV file using EWS and Powershell

Somebody asked about this last week and while I have a lot of EWS scripts that do access the Calendar I didn't have a simple example that just exported a list of the Calendar events with relevant information to a CSV file so here it is. I've talked on this one before in this howto  but when you query the calendar folder using EWS you need to use a CalendarView which will expand any recurring appointments in a calendar. There are some limits when you use a calendarview in that you can only return a maximum of 2 years of appointments at a time and paging will limit the max number of items to 1000 per call. So if you have a calendar with a very large number of appointments you need to break your query into small date time blocks. In this example script I'm just grabbing the next 7 days of appointments if you want to query a longer period you need to adjust the following lines (keeping in mind what I just mentioned) #Define Date to Query $StartDate = (Get-Date) $EndDate

Downloading a shared file from Onedrive for business using Powershell

I thought I'd quickly share this script I came up with to download a file that was shared using One Drive for Business (which is SharePoint under the covers) with Powershell. The following script takes a OneDrive for business URL which would look like https://mydom-my.sharepoint.com/personal/gscales_domain_com/Documents/Email%20attachments/filename.txt This script is pretty simple it uses the SharePoint CSOM (Client side object Model) which it loads in the first line. It uses the URI object to separate the host and relative URL which the CSOM requires and also the SharePointOnlineCredentials object to handle the Office365 SharePoint online authentication. The following script is a function that take the OneDrive URL, Credentials for Office365 and path you want to download the file to and downloads the file. eg to run the script you would use something like ./spdownload.ps1 ' https://mydom-my.sharepoint.com/personal/gscales_domain_com/Documents/Email%20attachments/filena

Writing a simple scripted process to download attachmentts in Exchange 2007/ 2010 using the EWS Managed API

Every complicated thing in life is made up of smaller simpler building blocks, when it comes to writing a script (or any code really) the more of these little building blocks you have to figure out the more the process of solving a problem can become bewildering. The Internet generally provides you with lots of half eaten sandwiches of information something someone else has taken a bite out but a lot of the time half done, and as with any code its usefulness declines over time as new and better API's and methods are derived. In this post I'm going to go through a simple scripted process that hopefully covers a few more of these smaller building blocks that you might face when asked to come up with a simple costless solution to perform an automated business function with a script. So the process im going to look at is one that comes up a lot and that is you have an Email that comes into to certain mailbox every day with a certain subject in my case "Daily Export" this
All sample scripts and source code is provided by for illustrative purposes only. All examples are untested in different environments and therefore, I cannot guarantee or imply reliability, serviceability, or function of these programs.

All code contained herein is provided to you "AS IS" without any warranties of any kind. The implied warranties of non-infringement, merchantability and fitness for a particular purpose are expressly disclaimed.