Monday, February 21, 2011

Exporting and Uploading Mailbox Items using Exchange Web Services using the new ExportItems and UploadItems operations in Exchange 2010 SP1

Two new EWS Operations ExportItems and UploadItems where introduced in Exchange 2010 SP1 that allowed you to do a number of useful things that where previously not possible using Exchange Web Services. Any object that Exchange stores is basically a collection of properties for example a message object is a collection of Message properties, Recipient properties and Attachment properties with a few meta properties that describe the underlying storage thrown in. Normally when using EWS you can access these properties in a number of a ways eg one example is using the strongly type objects such as emailmessage that presents the underlying properties in an intuitive way that's easy to use. Another way is using Extended Properties to access the underlying properties directly. However previously in EWS there was no method to access every property of a message hence there is no way to export or import an item and maintain full fidelity of every property on that item (you could export the item as an EML but this doesn't provide any fidelity of the properties on item).

Now with these two new operations there is a method of exporting and uploading items and maintaining all the Mapi properties. The only real restriction is by default the maximum import payload is 25MB of base64 encoded data but these setting can be modified in the web.config file. The export/import format that these two operations use in a stream format which is a stream of all the Exchange properties separated with meta properties to represent the different property collections on the Item. This format while it bares some similarity to TNEF and Compound Message format (MSG) is not the same.

There are no methods in the EWS Managed API to use these operations so you need to use EWS Proxy code or just raw SOAP when writing applications or script to use this. However you do need the EWSid's of the Items and Folder to upload or export items and the Managed API is the easiest way of getting these. For this post I've create a sample of exporting the last received item in the Inbox first using the EWS Managed API to find the last item and then use raw SOAP to export the items.

I posted a downloadable copy of the following EWS Managed API script here the code itself looks like

$MailboxName = "mailbox@domain.com"
$fileName = 'c:\exp.fts'
$cred = New-Object System.Net.NetworkCredential("username@domain.com","password")

$dllpath = "C:\Program Files\Microsoft\Exchange\Web Services\1.1\Microsoft.Exchange.WebServices.dll"
[void][Reflection.Assembly]::LoadFile($dllpath)
$service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP1)
$service.TraceEnabled = $false

$windowsIdentity = [System.Security.Principal.WindowsIdentity]::GetCurrent()
$sidbind = "LDAP://<SID=" + $windowsIdentity.user.Value.ToString() + ">"
$aceuser = [ADSI]$sidbind
$service.Credentials = $cred
$service.autodiscoverurl($MailboxName,{$true})

$folderid = new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox,$MailboxName)

$view = New-Object Microsoft.Exchange.WebServices.Data.ItemView(1)
$findResults = $service.FindItems($folderid,$view)
$itemid = $findResults.Items[0].Id.Uniqueid

$expRequest = @"
<?xml version="1.0" encoding="utf-8" ?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages">
<soap:Header>
<t:RequestServerVersion Version="Exchange2010_SP1" />
</soap:Header>
<soap:Body>
<m:ExportItems>
<m:ItemIds>
<t:ItemId Id="$itemId"/>
</m:ItemIds>
</m:ExportItems>
</soap:Body>
</soap:Envelope>
"@
$mbMailboxFolderURI = New-Object System.Uri($service.url)
$wrWebRequest = [System.Net.WebRequest]::Create($mbMailboxFolderURI)
$wrWebRequest.KeepAlive = $false;
$wrWebRequest.Headers.Set("Pragma", "no-cache");
$wrWebRequest.Headers.Set("Translate", "f");
$wrWebRequest.Headers.Set("Depth", "0");
$wrWebRequest.ContentType = "text/xml";
$wrWebRequest.ContentLength = $expRequest.Length;
$wrWebRequest.Timeout = 60000;
$wrWebRequest.Method = "POST";
$wrWebRequest.Credentials = $cred
$bqByteQuery = [System.Text.Encoding]::ASCII.GetBytes($expRequest);
$wrWebRequest.ContentLength = $bqByteQuery.Length;
$rsRequestStream = $wrWebRequest.GetRequestStream();
$rsRequestStream.Write($bqByteQuery, 0, $bqByteQuery.Length);
$rsRequestStream.Close();
$wrWebResponse = $wrWebRequest.GetResponse();
$rsResponseStream = $wrWebResponse.GetResponseStream()
$sr = new-object System.IO.StreamReader($rsResponseStream);
$rdResponseDocument = New-Object System.Xml.XmlDocument
$rdResponseDocument.LoadXml($sr.ReadToEnd());
$Datanodes = @($rdResponseDocument.getElementsByTagName("m:Data"))
if ($Datanodes.length -ne 0){
$Data = [System.Convert]::FromBase64String($Datanodes[0].'#text')
$fsFileStream = new-object system.io.filestream $fileName, ([io.filemode]::create), ([io.fileaccess]::write), ([io.fileshare]::none)
$fsFileStream.Write($Data, 0, $Data.Length);
$fsFileStream.Close();
}

27 comments:

Ryan said...

I'm trying to get a script to accept credentials previously stored in an attribute by using a -Credential argument, but when doing so it just returns the line "System.Management.Automation.PSCredential" instead of the credentials. If I just set a pre-named attribute that is always used by the script it works fine, though. Do you know how I would do this?

Also, I've only gotten EWS to accept credentials in the form of my SAM account name, is that expected?

Glen said...

PSCredential can't be used directly in EWS it needs to be a Network credential or web credential. You should be able to unwind the PSCredential something like http://poshcode.org/474. Generally i would use the account UPN rather then the sam account name this means you dont need to worry about using a doamin as you would with Samaccountname.

grapkulec said...

you wrote that exported items can be saved in files but does it mean that they could be used as an alternative to MSG files?

in our project MSG files are used for preserving message's body formatting and details like sender, subject, recipients, etc. what is great in case of MSG is that you can such file in Outlook. is it possible to use files with exported items in similar manner?

Glen said...

MSG files have nothing to do with Exchange these are a Office file format associated with Outlook. Underlying they are serialized representation of all the Items Mapi properties. The Fast transfer Stream is an Exchange format and is also a serialized representation of Items Mapi properties but you will never be able to read this in Outlook for example. Microsoft are only supporting using the FTS stream as an Opaque stream to transfer data between two Exchange server running the same Store version (eg SP and Rollup). So while you could store the Items as files that could be re-uploaded later etc but you could only guarantee that you could reupload them to a Exchange server that is running the same version (SP/Rollup) as where the Items where downloaded from. Eg the format of FTS may change between Service Packs and rollups because its not supported as storage format.

Cheers
Glen

grapkulec said...

Glen, I think you just saved us from a huge pitfall because we were about to start investigation how we could use files created from ExportItems stream to develop our own MSG-like storage.

So, as far as we know there is no other way than write our own message serialization to MSG file to preserve original message formatting and this of course is neither recommended nor supported by Microsoft way of doing things and basically we are asking for a lot of problems.

Glen said...

Yes but the one counter to that i have to say is that if you want to serialize a Store Item using EWS then exportItems is the only way you achieve because this is the only method the EWS provides of getting access to raw recipient and attachment properties. The support side isn't great but if your trying to do serialization over EWS it maybe something you have to live with until MS come up with a better solution it just put more pressure on the support side of the coding process (but hey it keeps us developers in a job :)

grapkulec said...

I watched presentation on Fast Transfer that you linked and it seems that data in this streams are more or less easy to extract so maybe it would be not so bad idea after all to use them for providing data into MSG. maybe easier than just preparing those data from properties of an items encapsulated in EWS Managed API.

after all we rely on Microsoft in so many places that one more functionality where some future change on their side can break our side makes a little difference :)

anyway, thank you very much for your answers and explanations. it is a shame that msdn documentation on EWS is so scarce and minimalistic.

Chris Padgett said...

Hi Glen

I don't believe that the above script runs successfully if the service user ($cred) doesn't "own" the mailbox ($MailboxName) from which items are to be exported.

Do you know if you can export items from a mailbox that isn't yours?

Does it require impersonation or a specific permission?

Regards,

Chris

Gagan Bhatnagar said...

Hi Glen,

I am using Exportitems to backup items like mails, appointments from Exchange online server and store the stream on disk. Is there a way to find out the size of the base64 encoded stream that we receive from Exchange server beforehand. My client needs to know the size of the stream before he tries to fetch it from Exchange server. Also, what is the reason that size of the actual mail seen using EWS is less than the size of the base 64 stream obtained using EWS for the same mail.

Glen Scales said...

>> Is there a way to find out the size of the base64 encoded stream that we receive from Exchange server beforehan

Not the exact size it should be roughly the size of the message, the reason it maybe bigger then the message is that its a transfer format designed to reconstitute the message on a Exchange server so its has overhead information vs just stored. Sizes are always variable eg if you export a Mailbox to PST in Oulook the PST is always much larger then the Mailbox.

Cheers
Glen

Gagan Bhatnagar said...

Hi Glen,

Can we extract attachment files(s) from the encoded data stream I receive for a mail item from Exchange using ExportItems.

Thanks,
Gagan

Glen Scales said...

If you want to extract Attachmenst then you should use EWS GetAttachment methods. The Export and UploadItems operations are meant for Transfer of data between servers not for storage. If you want to extract attachments from an Item that is nolonger on Exchange you should restore it first (using uploadItems) and then access it using GetAttachment this will ensure everything is converted correctly .

Cheers
Glen

Gagan Bhatnagar said...

Thanks Glen !

Gagan Bhatnagar said...

Hi Glen,
Can I use the EWS Fast Stream parser (http://ewsftstream.codeplex.com/ ) to extract attachments from the decoded base 64 data stream of an item which is obtained using ExportItems. Can this parser work for Exchange 2010 as well as 2013 and online ?
Thanks,
Gagan

Glen Scales said...

The only method that I can recommend that would be supported and consistent (which is really important if your talking about backups) is to first upload the Item to Exchange and then use the EWS Operations attachment operations.

Cheers
Glen

Gagan Bhatnagar said...

Hi Glen,

I am facing an issue. I backed up an item in Exchange Online mailbox with 6 attachments of size 2.5 MB each successfully using Exportitems EWS api
The total size of the mail item was 15 MB

when I restore the item to Exchange Online using UploadItems EWS api, I received the following error:


The underlying connection was closed: An unexpected error occurred on a send.

Is it failing due to EWS throttling limits. How can i determine the maximum size of a mail item that can be restored without any errors.

Thanks,
Gagan

Glen Scales said...

Make sure you set the X-AnchorMailbox header to the mailbox your uploading the message to http://msdn.microsoft.com/en-us/library/office/dn458789(v=exchg.150).aspx . Otherwise poor networks connections can cause that, I get that occasionally on my crappy residential ADSL link there isn't much you can do other then retry. You might want to try testing it from a Azure VM then you know you have a good link

Cheeers
Glen

Gagan Bhatnagar said...

Thanks Glen, I added X-AnchorMailbox header but it did not work. However, as per your suggestion I tried this on a high connection (10 Mbps upload speed) and it worked. Thanks once again for your answer. I may not have resolved the without your reply.
Regards, Gagan

Gagan Bhatnagar said...

Glen, Is it possible to somehow manage uploads on a slow connection. My client has a relatively slow connection where this fails. Thanks, Gagan

Glen Scales said...

I'm not sure it depends why the upload failed eg if the link is dropping or your getting a lot of packloss etc the best thing you can probably do is catch the error and retry. Maybe look at using different Class to do the upload eg HttpWebRequest has more options then using the proxy classes. Whenever I've had that issue retrying the same upload will usually fix it. Maybe test uploading large attachments in OWA and then try to workout if it uses different headers etc.

Cheers
Glen

Gagan Bhatnagar said...

Glen, I have another query.

I am working on a C# project that requires Calendar events to be restored to a target mailbox. When I restore a Calendar event with attendees to an alternate target mailbox, using Exchange Web Service Upload Items api, Exchange tries to send mails to all attendees of the Calendar event.

This is undesirable behavior since restoration of a past event should not result in mails being sent to attendees. How can I restrict Exchange from sending these mails.

Also, the ownership pf the calendar event should get changed from the original mailbox user to the target mailbox user. Right now, on restoring calendar events to an alternate mailbox in a different domain, Exchange online fails with an error "You do not have the permission to send the message on behalf of the specified user."

Can you give your thoughts on this issue.

Thanks, Gagan

Glen Scales said...

I'd say your getting Mails to the attendees when you update the appointment rather then upload it, I've never seen that happen on an upload.

My suggestion is if you need to change the Organizer property you will need to use MAPI to do this. Once an appointment is in a Mailbox you won't be able to change this. So you'll probably need to export it out make the change then import it back in using MAPI.

Glen

Gagan Bhatnagar said...

You are right..After uploading the item, I had to update it for some reason and that was causing this issue. There's a parameter I used that stops notification on Update. Thanks.

Gagan Bhatnagar said...

I resolved this issue by setting Keepalive as false for the webrequest i was making to uploaditems. Thanks.

Gagan Bhatnagar said...

Hi Glen,

I am facing a strange issue where you help is needed.
I have written a backup solution that backups Exchange online items on disk using ExportItems EWS method.
Once the mailbox backup is complete, I delete the mailbox. Next, I restore the mailbox, I use Windows Azure AD powershell cmdlets to Create the new mailbox in Exchange Online. When the mailbox is created. I query Exchange Online using Get-SearchableMailboxes EWS method to check if the mailbox
exists.

Next, I use Uploaditems (EWS method) to restore items to this new mailbox. But before restoring items I check if the items to be restored already exist in the target mailbox using FindItems()
If they exist I do not restore them. Surprisingly, the items are always found to be existing in the mailbox which is contrary to the expected results. Is it possible that EWS is returning these results from the deleted mailbox ?


I am facing a similar issue for same solution for Exchange On-premises server 2013.
1. If I use adim credentials to restore items after the new mailbox is created I get an Error - Impersonation failed.
The first attempt to restore mailbox from backup after deletion of the mailbox is always successful.
However, the next attempts to restore mailbox from backup after deletion of the newly created mailbox always fails with the error Impersonation failed.

2. If I use the newly created mailbox credentials to restore items (i.e. without impersonation), EWS finds that the items already exist in the newly created mailbox. Ideally the newly created mailbox should have no items that were backed up. It is a similar behavior to what we found in Exchange Online.

Can you please suggest a solution or a workaround to address this issue.

Regards,
Gagan

Anonymous said...

Hi Glen

Have you pulled the ewsftsstream codeplex project?

Thanks
Michel

Praveen Giri said...
This comment has been removed by the author.