Wednesday, July 19, 2017

Using the Office365/Exchange 2016 REST API for working with Mailbox and OneDrive files/attachments

In my last post I demonstrated how to rewrite a simple download process script from using EWS to using the new REST API in Office365. This was an example of an Automation process that I first used in the early 2000's as back then email offered a ubiquity for a simple data transfer that even today is hard to do in some enterprises which is why that post has been so popular over the years.

In this post I'm going to show you how you can make use of other Office365 endpoints available in the Graph API to basically take this old process and start improving on it. The particular Endpoint I'll be focusing on in this post is OneDrive https://dev.onedrive.com/ (or One Drive for business). I'll also be using my Exch-Rest Module and taking advantage of all the existing code I've written for the interacting with  Exchange Online to interact with the OneDrive endpoint. And that really is the one great thing about the new Graph Endpoint is that all I've needed to do to add OneDrive functionally to my module is to create some new functions to deal with the specific addressing differences between Mail and One Drive while the underlying Auth, Get and Post function are the same (so speed and agility in marketing fluff and less hours recoding in real speak).

Basics of Interacting with One Drive

Authentication : To use One Drive from the Exch-Rest Module the application registration you have been using for your Mail scripts needs to be granted the OneDrive Permission Grants in the Azure portal https://dev.onedrive.com/auth/graph_oauth.htm. Once the grants have been added they will be then returned when AccessToken is requested, so basically you can use the same AccessToken for accessing Email and OneDrive without the need to authenticate again.

Paths and Drives:
 Each user has a default OneDrive drive which you can access using the /drive/root path . If you know the Id of the file you want to access then you can use the following Path drive/items/{item-id} to access it otherwise if you know the relative path (eg folder path/filename) then this can be used like /drive/root:/path/to/file to access the file. (this is a pretty basic explanation but the documentation cover this in more depth https://dev.onedrive.com/ )

 This will make more sense when you look at an example so lets start

Modern Attachments aka Reference Attachments

Modern Attachments is a new feature introduced into Exchange,Office365 and Outlook 2016 that was a kind of inbuilt OneDrive and Exchange integration feature. In EWS and REST these type of attachments are called reference attachments . Within the REST API currently Reference attachments are supported in the v2.0 Outlook endpoint and the Beta endpoint in the current Graph API. I've added support for sending reference attachments in Exch-Rest (v1.9) so an example of sending a reference attachment (eg a one drive file) using the Graph Beta Endpoint using the module would look like

Import-Module Exch-Rest -Force
$MailboxName = "gscales@datarumble.com"
$OneDriveFile = "/test/test2/fileName.zip"
$DownloadDirectory = "c:\temp"
##Get the Access Token
$AccessToken =  Get-AccessToken -MailboxName $MailboxName  -ClientId 5471030d-f311-4c5d-91ef-74ca885463a7 -redirectUrl "urn:ietf:wg:oauth:2.0:oob" -ResourceURL graph.microsoft.com -Beta 
##Get OneDrive DownloadURI
$OneDriveAttachmentToSend = Get-OneDriveItemFromPath -MailboxName $MailboxName -AccessToken $AccessToken -OneDriveFilePath $OneDriveFile
$rtArray = @()
$rtArray += (New-referanceAttachment -Name $OneDriveAttachmentToSend.Name -SourceUrl $OneDriveAttachmentToSend.webUrl -Permission Edit)
Send-MessageREST -MailboxName $MailboxName  -AccessToken $AccessToken -ToRecipients @(New-EmailAddress -Address glenscales@yahoo.com) -Subject "Daily Send" -Body "See Attached" -ReferanceAttachments $rtArray
 Full Source Available here

In this example the Get-OneDriveItemFromPath function in the module is being used to get the necessary Reference Attachment properties (Name and WebURL) from the OneDrive file using the relative onedrive path. The New-referanceAttachment function creates a custom object that then Send-MessageREST function will be able to construct the necessary REST message to send.

Downloading a File from One Drive and Sending it as a normal Attachment

Modern Attachments are great but a lot of the time you will just need to send a particular one drive file as a regular attachment. In this case we need some more code to download the file you want to send from One Drive so it can then be Sent normally eg here is an example of doing this

Import-Module Exch-Rest -Force
$MailboxName = "gscales@datarumble.com"
$OneDriveFile = "/test/test2/fileName.zip"
$DownloadDirectory = "c:\temp"
##Get the Access Token
$AccessToken =  Get-AccessToken -MailboxName $MailboxName  -ClientId 5471030d-f311-4c5d-91ef-74ca885463a7 -redirectUrl "urn:ietf:wg:oauth:2.0:oob" -ResourceURL graph.microsoft.com 
##Get OneDrive DownloadURI
$OneDriveAttachmentToSend = Get-OneDriveItemFromPath -MailboxName $MailboxName -AccessToken $AccessToken -OneDriveFilePath $OneDriveFile
$DownloadFileName = $DownloadDirectory + "\" + $OneDriveAttachmentToSend.Name
Invoke-WebRequest -Uri $OneDriveAttachmentToSend.'@microsoft.graph.downloadUrl' -OutFile $DownloadFileName
Send-MessageREST -MailboxName $MailboxName  -AccessToken $AccessToken -ToRecipients @(New-EmailAddress -Address user@domain.com) -Subject "Daily Send" -Body "See Attached" -Attachments @($DownloadFileName)
Full Source Available here

In this example the Get-OneDriveItemFromPath function is used again to get the details of the One Drive file using its friendly relative path. The property that is used this time is the @microsoft.graph.downloadUrl' which is a preautheticated short term download URL that is then used in Invoke-WebRequest to download the file from OneDrive to the local file system so it can just then be sent as a regular file attachment in Send-MessageREST.

Saving an Attachment from a Message to One Drive (Simple scripted download Attachment script modification)

In the last example I'm looking at modifying my previous script from this post so instead of saving the attachment to disk it instead saves them to One Drive. I've used the simple upload method so this would mean the attachment size is limited to 4 MB maximum (you would need to use another method if you have files that are larger then this that need uploading).

Import-Module Exch-Rest -Force

#Import-Module Exch-Rest -Force
$MailboxName = "gscales@datarumble.com"
$Subject = "Daily Export"
$ProcessedFolderPath = "\Inbox\Processed"
$OneDriveUploadFilePath = "/test" 

##Get the Access Token
$AccessToken =  Get-AccessToken -MailboxName $MailboxName  -ClientId 5471030d-f311-4c5d-91ef-74ca885463a7 -redirectUrl "urn:ietf:wg:oauth:2.0:oob" -ResourceURL graph.microsoft.com  
##Search the Inbox
$Filter = "IsRead eq false AND HasAttachments eq true AND Subject eq '" + $Subject + "'"
$Items = Get-FolderItems -MailboxName $MailboxName -AccessToken $AccessToken -FolderPath \Inbox -Filter $Filter
if($Items -ne $null){
   if($Items -is [system.array]){
         Write-Host ($Items.Count.ToString() + " Items Found ")
   }
   else{
        Write-Host "Found 1 item"
   }
   foreach ($item in $Items) {
        Write-Host ("Processing Item received " + $Item.receivedDateTime)
        $item
        Get-Attachments -MailboxName $MailboxName -ItemURI $item.ItemRESTURI -MetaData -AccessToken $AccessToken | ForEach-Object{
            $attach = Invoke-DownloadAttachment -MailboxName $MailboxName -AttachmentURI $_.AttachmentRESTURI -AccessToken $AccessToken
            $attachBytes = [System.Convert]::FromBase64String($attach.ContentBytes)   
            $OneDriveFilePath = $OneDriveUploadFilePath + "/" + $attach.Name.ToString()
            Invoke-UploadOneDriveItemToPath -AccessToken $AccessToken -MailboxName $MailboxName -OneDriveUploadFilePath $OneDriveFilePath -FileBytes $attachBytes 
            write-host ("uploaded " + $OneDriveFilePath + " to one drive")
        }
        $UpdateProps = @()
        $UpdateProps += (Get-ItemProp -Name IsRead -Value true -NoQuotes)
        Update-Message -MailboxName $MailboxName -AccessToken $AccessToken -ItemURI $item.ItemRESTURI -StandardPropList $UpdateProps
        Move-Message -MailboxName $MailboxName -ItemURI $item.ItemRESTURI -TargetFolderPath $ProcessedFolderPath -AccessToken $AccessToken                
   }
  
}
else{
    Write-Host "No Item found"
}
Full Source Available here

The main change in this script is in the attachment processing code where the Invoke-UploadOneDriveItemToPath is used. This function does a simple upload of the file to OneDrive https://dev.onedrive.com/items/upload_put.htm so the total size of the file can't exceed 4MB.

All the scripts in this blog require version 1.9 of the Exch-Rest module from the PowerShell gallery https://www.powershellgallery.com/packages/Exch-Rest/1.9

The OneDrive API offers a lot more functionality that I'll share with more in future posts.

Wednesday, July 05, 2017

Simple scripted download Attachment using the Graph or Outlook Rest API in Office365

This is a rewrite of one of the more popular EWS posts (original post) on my blog about creating a scripted process that would download attachments from an email with a particular subject line in the Inbox and then mark that email read and move it to another folder in the Mailbox. In this post I'm going to go through a direct one to one rewrite of this script and talk about the comparisons between how you used the operations in EWS and now how you can do the same thing using the Graph or Outlook REST API. In another post I'll show an enhanced version that allows you to use the Graph API to save the attachment into another Graph endpoint such as OneDrive or SharePoint.

For this script I'm using the Exch-Rest Module I'm currently building which is available on the powershellgallery https://www.powershellgallery.com/packages/Exch-Rest and the source from Github https://github.com/gscales/Exch-Rest. For this script I had to overcome an issue with the ConvertFrom-Json Cmdlet which has an issue once the JSON payload gets over 2 MB so an alternate method in the REST code was needed to allow for attachment downloads of over 2MB.

Step 1: Loading the Module, Setting up variables for the script and Getting the Access Token

Import-Module Exch-Rest -Force
$MailboxName = "user@domain.onmicrosoft.com"
$Subject = "Daily Export"
$ProcessedFolderPath = "\Inbox\Processed"
$downloadDirectory = "c:\temp"
##Get the Access Token
$AccessToken =  Get-AccessToken -MailboxName $MailboxName  -ClientId 5471030d-f311-4c5d-91ef-74ca885463a7 -redirectUrl "urn:ietf:wg:oauth:2.0:oob" -ResourceURL graph.microsoft.com         

The first step in the scripted process is to load the module then setup some variables to hold which mailbox to access, the subject of the Message to search for and the folder to move the message to once its has been processed. Then finally we have the code to get the oauth AccessToken,with getting the Access Token there are few different options described here you need to choose the most appropriate one for you depending on how you are going to implement this in your environment. The EWS equivalent here would have been loading the Managed API, authenticating and doing an Autodiscover

Step 2: Search for the Items in the Inbox that are Unread, Have Attachments and the Subject line "Daily Export"
 
$Filter = "IsRead eq false AND HasAttachments eq true AND Subject eq '" + $Subject + "'"
$Items = Get-FolderItems -MailboxName $MailboxName -AccessToken $AccessToken -FolderPath \Inbox -Filter $Filter
In the REST API like EWS there are two different ways of searching for items
  • Filter
In EWS this was done using the SearchFilters Class (or a restriction in EWS proxy code) in REST this is done using the OData parameter $filter. The underlying mechanism the Exchange store uses for both EWS and REST to filter items and returned results is based on Restrictions (I still like the explanations in this article as to how they work https://technet.microsoft.com/en-us/library/cc535025(v=exchg.80).aspx) . Using Filters gives you the most flexibility when it comes to finding particular items based on particular properties but this does come at the cost of performance. You should try not to over complicate your filters too much as like any database your performance and application will suffer if the filters are written poorly with no regards to data structure or quantity.
  • Content Index Searches
In EWS a content index search could be done using AQS (Advance Query Syntax) and the QueryString Element in the FindItem operation. In REST you use the $Search OData parameter https://msdn.microsoft.com/en-us/office/office365/api/complex-types-for-mail-contacts-calendar#OdataQueryParams and the same AQS syntax you use in EWS. These searches should yield much better performance when you are searching folders with much higher Item counts because they utilize the Content indexes. There are limitations around using Search however because only certain properties are indexed and available for use (see https://technet.microsoft.com/en-us/library/jj983804(v=exchg.150).aspx) and there can be a delay in Indexes being updated.

I've used the $filter parameter in my above example which reflects the SearchFilter used in EWS in the original post but Search could also be used in this example.

Step 3: Processing the Attachments on the Items

Get-Attachments -MailboxName $MailboxName -ItemURI $item.ItemRESTURI -MetaData -AccessToken $AccessToken | ForEach-Object{
$attach = Invoke-DownloadAttachment -MailboxName $MailboxName -AttachmentURI $_.AttachmentRESTURI -AccessToken $AccessToken
$fiFile = new-object System.IO.FileStream(($downloadDirectory  + "\" + $attach.Name.ToString()), [System.IO.FileMode]::Create)
$attachBytes = [System.Convert]::FromBase64String($attach.ContentBytes)   
$fiFile.Write($attachBytes, 0, $attachBytes.Length)
$fiFile.Close()
write-host ("Downloaded Attachment : " + (($downloadDirectory + "\" + $attach.Name.ToString())))
 Like EWS when you enumerate the Items in folder using REST it won't return all the information about the Item for performance reasons. So like EWS where you would then use a GetItem request to get the Metadata information about attachments in REST you need to do a simular thing before you can proceed to downloading the attachment. In the module I have the Get-Attachments function for returning the MetaData about attachments (this writes a custom AttachmentRESTURI property into the results). Then the Invoke-DownloadAttachment function does the downloading of the attachments content. If the message has more then one attachment this is where you would modify the code to do additional processing (eg restriction on FileName, FileType etc).

Step4: Marking the Item as Read

$UpdateProps = @()
$UpdateProps += (Get-ItemProp -Name IsRead -Value true -NoQuotes)
Update-Message -MailboxName $MailboxName -AccessToken $AccessToken -ItemURI $item.ItemRESTURI -StandardPropList $UpdateProps
In EWS to mark an Item as read you would update the IsRead Strongly typed property and then call UpdateItem operation. In REST it is much the same procedure except we have some code that defines the Property you want to update in JSON (in this case the isRead property) and then it uses a HTTP Patch request to send the update to the server along with the URI of the Item you are updating.

Step5: Moving the Item to another folder

Move-Message -MailboxName $MailboxName -ItemURI $item.ItemRESTURI -TargetFolderPath $ProcessedFolderPath -AccessToken $AccessToken     
 In EWS to move an Item to another folder you use the MoveItem operation, in REST its fairly simular you invoke a move by sending the URI of the Item with /move appended to the end with a JSON payload of destination FolderId https://msdn.microsoft.com/en-us/office/office365/api/mail-rest-operations#move-or-copy-messages which is what Move-Message function does in the Module.

that's pretty much it I've posted the full code for this sample on GitHub here https://github.com/gscales/Exch-Rest/blob/master/Samples/simpleAttachDownload.ps1