Message Tracking from an operation perspective is one of the most useful windows you have into what's happening on your Exchange server. Used skillfully these logs can tell you firstly that your server is operating correctly eg you are receiving messages and people are sending messages, how much data is coming in, going between, and going out of your network via Email. And when the inevitable problem happens and a message is delayed or not delivered message tracking becomes one of the most import tools for diagnosis.
But Message tracking logs are only a fraction of the information that is contained on a message at the point of time it was traversing the Transport pipeline. But if you combine Message Tracking with an Exchange Store API like Exchange Web Services you suddenly have a very powerful tool that can give you an unlimited amount of flexibility to audit,analysis and take action on your environment. Okay I maybe getting a little over enthusiastic but this is one area i think is a little underutilized because of a little snow blindness you have your sysadmins in one camp who's realm is Exchange Message Tracking and you have a number of good products that can produce increasing complex reports. And the other side your developers who take advantage of the store Api to build item based applications and content and archiving applications. If you bring these two worlds together we can start using the Message Tracking logs as an active tool that can help both sides of this equation for example you see a large message in your tracking logs that your worried may breach an important company policy. Now typically in the Tracking logs you would have the size of the message and subject and who sent it along with one other important piece of information the Internet MessageID. Typically now if an admin wanted to check that message they would need assign an account rights to the mailbox that send or received the message and then using outlook open the mailbox and locate that message in the users mailbox which is a time consuming process. Exchange Web Services has a number of key things that can help simplify this process.
What are the catches to this method firstly in Exchange one user can send as another if they have permissions and in this case although the log files will reflect the sender address as the mailbox that sent the item that actual sent email will be located in another mailbox. Also the dumpster dive method doesn't account for messages in deleted folders.
Okay let get to some EWS code to search the inbox for a particular message ID for those non developers out there hang in their I'll give you a powershell method later.
FindItemType fiFindItemRequest = new FindItemType();
fiFindItemRequest.Traversal = ItemQueryTraversalType.Shallow;
ItemResponseShapeType ipItemProperties = new ItemResponseShapeType();
ipItemProperties.BaseShape = DefaultShapeNamesType.IdOnly;
fiFindItemRequest.ItemShape = ipItemProperties;
DistinguishedFolderIdType dfDfolder = new DistinguishedFolderIdType();
dfDfolder.Id = DistinguishedFolderIdNameType.inbox;
DistinguishedFolderIdType[] faFolderIDArray = new DistinguishedFolderIdType[1];
faFolderIDArray[0] = new DistinguishedFolderIdType();
faFolderIDArray[0] = dfDfolder;
fiFindItemRequest.ParentFolderIds = faFolderIDArray;
Restriction Type ffRestriction = new Restriction Type();
IsEqualToType ieToType = new IsEqualToType();
PathToUnindexedFieldType miMessageID = new PathToUnindexedFieldType();
miMessageID.FieldURI = UnindexedFieldURIType.messageInternetMessageId;
FieldURIOrConstantType ciConstantType = new FieldURIOrConstantType();
ConstantValueType cvConstantValueType = new ConstantValueType();
cvConstantValueType.Value = "messageid@domain...";
ciConstantType.Item = cvConstantValueType;
ieToType.Item = miMessageID;
ieToType.FieldURIOrConstant = ciConstantType;
ffRestriction.Item = isotype;
fiFindItemRequest.Restriction = ffRestriction;
FindItemResponseType frFindItemResponse = esb.FindItem(fiFindItemRequest);
Okay hopefully i haven't lost anyone after that but to make this useful for those that can't put the above code into Visual Studio I've wrapped up a modified version of the above code as well as another method to Recurse ever folder in a mailbox,cater for dumpster diving, exporting to EML and also download attachments into a Class library (DLL) that can be loaded and used with a few extra lines in Powershell. So as a Sysadmin with this you can do things like use Message tracking to find messages that where over 5 MB today and then pipe them into this DLL to download all the attachments or export the messages to a directory. Or just produce a more detailed report on the content of the messages and attachments. Or export all the communication between a particular domain or on a particular subject. Or pretty much anything else you can put your mind to (or all those crazy requests you get from HR department).
To do a recursive search of a mailbox with EWS you need to issue a separate Finditems request for each folder as there is no Deep Traversal for find items. To get the Folder hierarchy you can however do a Deep Traversal which will return a list of all folders within a Mailbox. Matt Stehle recently posted this which describe how to get the best performance out of this type of operation.
This library supports Autodiscovery based on the Email address you enter and also support both Impersonation and Delegation authentication models. It also supports hardcoding the username, password and domain as well as the EWS/CAS URL which allows you to use the library from a remote machine anywhere in the network without it needing to be a member of the domain.
To use the library from powershell you basically need to first load the dll eg
[void][Reflection.Assembly]::LoadFile("C:\temp\EWSUtil.dll")
Then you can use the objects that are defined within this class the first thing you need to do is create a EWS connection (this is a custom class I've created within the Class library that contains a Exchange Service Binding for EWS). The parameters you pass into the objects creation affect what authentication is used (eg impersonation or delegation) and also whether autodiscover is used or not. I haven't included any overloads so all the parameters are mandatory you just need to pass in $null if you don't want to use some. So the parameters for the EWSConnection object.
$casURL = "https://" + $servername + "/EWS/Exchange.asmx"
$ewc = new-object EWSUtil.EWSConnection("user@domamain.com",$false, "username", "password", "domain", $casURL )
The library returns a generic List of objects that matchs the restiction which depending on the code you use can be the messageID or to be as multi use as possible you can use any of the unindexed properties such as the Subject Etc.
The following is a sample that uses firstly the get-messagetrackinglogs cmdlet to retrieve the MessageID for any messages that are over 5 MB that have been recieved today in the last 2 hours. Then it looks at the recipient address and verfies that they are local accounts and then uses get-user to make sure you are working with the Primary Email address for the mailbox and not a proxy address which would make this script fail. It then uses Impersonation and the currently logged on user credentials to search for the message in the inbox of the recipient and exports the message and any attachments to c:\export\message and c:\export\attachments respectivly.
[void][Reflection.Assembly]::LoadFile("C:\temp\EWSUtil.dll")
function getMessage($recpAddress){
$recp
$ascii = new-object System.Text.ASCIIEncoding
$ewc = new-object EWSUtil.EWSConnection($recpAddress,$true, "", "", "",$null)
$dType = new-object EWSUtil.EWS.DistinguishedFolderIdType
$dType.Id = [EWSUtil.EWS.DistinguishedFolderIdNameType]::inbox
$fldarry = new-object EWSUtil.EWS.BaseFolderIdType[] 1
$fldarry[0] = $dType
$randNumber = New-Object system.random
$prop = new-object EWSUtil.EWS.PathToUnindexedFieldType
$prop.FieldURI = [EWSUtil.EWS.UnindexedFieldURIType]::messageInternetMessageId
$messagelist = $ewc.FindMessage($fldarry,$prop,$_.MessageID,$false)
foreach ($message in $messagelist){
$baByteArray = [Convert]::FromBase64String($message.MimeContent.Value)
$emlMessage = $ascii.GetString($baByteArray)
$emlfile = new-object IO.StreamWriter(("c:\export\message\" + $message.Subject.Replace("#","").Replace(":","") + $mc+ ".eml"),$true)
$emlfile.WriteLine($emlMessage)
$emlfile.Close()
"Exported Message " + $message.Subject
$mc = $mc +1
if ($message.hasattachments){
"Exported Message to " + ("c:\message\" + $message.Subject + $mc + ".eml")
foreach($attach in $message.Attachments){
$ewc.DownloadAttachment(("c:\export\Attachments\" + $randNumber.next(1,1000) + $attach.Name.ToString()),$attach.AttachmentId);
"Downloaded Attachment : " + $attach.Name.ToString()
}
}
}
}
$servername = "servername"
$DomainHash = @{ }
get-accepteddomain ForEach-Object{
if ($_.DomainType -eq "Authoritative"){
$DomainHash.add($_.DomainName.SmtpDomain.ToString().ToLower(),1)
}
}
$dtQueryDT = [DateTime]::UtcNow.AddHours(-2)
$dtQueryDTf = [DateTime]::UtcNow
Get-MessageTrackingLog -Server $servername -ResultSize Unlimited -Start $dtQueryDT -End $dtQueryDTf -EventId "RECEIVE" where {$_.TotalBytes -gt 5242880} ForEach-Object{
foreach($recp in $_.recipients){
if ($recp -ne ""){
$recparray = $recp.split("@")
if ($DomainHash.ContainsKey($recparray[1])){
$vuser = get-user $recp
getMessage($vuser.WindowsEmailAddress)
}
}
}}
To make this script search the whole mailbox instead of just the inbox you would need to change the following two lines,
first you need to set the parent folder to the root of the mailbox
$dType.Id = [EWSUtil.EWS.DistinguishedFolderIdNameType]::msgfolderroot
And change the following line to RecurseFolder
$messagelist = $ewc.RecurseFolder($fldarry,$prop,$_.MessageID,$false)
If you wanted to do a dumpsterdive change the last parameter to $true
I'm really just scratching the surface of the cool things you can do with this and i could keep rabbiting on for days but I've got bring this to end. I've put a download of the lastest verion of the library along with the source for those that want to roll your own here the sample script can be downloaded from here. As i mentioned before this library is untested and highly experimental and only sutible for test enviroments.
But Message tracking logs are only a fraction of the information that is contained on a message at the point of time it was traversing the Transport pipeline. But if you combine Message Tracking with an Exchange Store API like Exchange Web Services you suddenly have a very powerful tool that can give you an unlimited amount of flexibility to audit,analysis and take action on your environment. Okay I maybe getting a little over enthusiastic but this is one area i think is a little underutilized because of a little snow blindness you have your sysadmins in one camp who's realm is Exchange Message Tracking and you have a number of good products that can produce increasing complex reports. And the other side your developers who take advantage of the store Api to build item based applications and content and archiving applications. If you bring these two worlds together we can start using the Message Tracking logs as an active tool that can help both sides of this equation for example you see a large message in your tracking logs that your worried may breach an important company policy. Now typically in the Tracking logs you would have the size of the message and subject and who sent it along with one other important piece of information the Internet MessageID. Typically now if an admin wanted to check that message they would need assign an account rights to the mailbox that send or received the message and then using outlook open the mailbox and locate that message in the users mailbox which is a time consuming process. Exchange Web Services has a number of key things that can help simplify this process.
- AutoDiscover : You can use the Email Address in the case of Message Tracking the Sender or Recipient Address to find the correct URL to use for EWS to open/query the mailbox that sent of received the message recorded in the log
- Impersonation : Impersonation allows you to Impersonate the Sender or Receiver of the message to access their mailbox and get the content. Impersonation rights can be granted at the Store or Mailbox level and alleviates the need to grant mailbox rights to an accounts to access mailboxes on a ad hoc basis
- Soft Deleted Traversal: If the user has deleted the item after they have sent or received the message then the message will be located in the dumpster of one of the folders in the Mailbox. A Soft deleted traversal can be used to search the dumpster of a folder to find the item. A limitation of EWS is that you can't actually access the item in the dumpster so another API such as MAPI needs to used to recover/export the item if necessary.
What are the catches to this method firstly in Exchange one user can send as another if they have permissions and in this case although the log files will reflect the sender address as the mailbox that sent the item that actual sent email will be located in another mailbox. Also the dumpster dive method doesn't account for messages in deleted folders.
Okay let get to some EWS code to search the inbox for a particular message ID for those non developers out there hang in their I'll give you a powershell method later.
FindItemType fiFindItemRequest = new FindItemType();
fiFindItemRequest.Traversal = ItemQueryTraversalType.Shallow;
ItemResponseShapeType ipItemProperties = new ItemResponseShapeType();
ipItemProperties.BaseShape = DefaultShapeNamesType.IdOnly;
fiFindItemRequest.ItemShape = ipItemProperties;
DistinguishedFolderIdType dfDfolder = new DistinguishedFolderIdType();
dfDfolder.Id = DistinguishedFolderIdNameType.inbox;
DistinguishedFolderIdType[] faFolderIDArray = new DistinguishedFolderIdType[1];
faFolderIDArray[0] = new DistinguishedFolderIdType();
faFolderIDArray[0] = dfDfolder;
fiFindItemRequest.ParentFolderIds = faFolderIDArray;
Restriction Type ffRestriction = new Restriction Type();
IsEqualToType ieToType = new IsEqualToType();
PathToUnindexedFieldType miMessageID = new PathToUnindexedFieldType();
miMessageID.FieldURI = UnindexedFieldURIType.messageInternetMessageId;
FieldURIOrConstantType ciConstantType = new FieldURIOrConstantType();
ConstantValueType cvConstantValueType = new ConstantValueType();
cvConstantValueType.Value = "messageid@domain...";
ciConstantType.Item = cvConstantValueType;
ieToType.Item = miMessageID;
ieToType.FieldURIOrConstant = ciConstantType;
ffRestriction.Item = isotype;
fiFindItemRequest.Restriction = ffRestriction;
FindItemResponseType frFindItemResponse = esb.FindItem(fiFindItemRequest);
Okay hopefully i haven't lost anyone after that but to make this useful for those that can't put the above code into Visual Studio I've wrapped up a modified version of the above code as well as another method to Recurse ever folder in a mailbox,cater for dumpster diving, exporting to EML and also download attachments into a Class library (DLL) that can be loaded and used with a few extra lines in Powershell. So as a Sysadmin with this you can do things like use Message tracking to find messages that where over 5 MB today and then pipe them into this DLL to download all the attachments or export the messages to a directory. Or just produce a more detailed report on the content of the messages and attachments. Or export all the communication between a particular domain or on a particular subject. Or pretty much anything else you can put your mind to (or all those crazy requests you get from HR department).
To do a recursive search of a mailbox with EWS you need to issue a separate Finditems request for each folder as there is no Deep Traversal for find items. To get the Folder hierarchy you can however do a Deep Traversal which will return a list of all folders within a Mailbox. Matt Stehle recently posted this which describe how to get the best performance out of this type of operation.
This library supports Autodiscovery based on the Email address you enter and also support both Impersonation and Delegation authentication models. It also supports hardcoding the username, password and domain as well as the EWS/CAS URL which allows you to use the library from a remote machine anywhere in the network without it needing to be a member of the domain.
To use the library from powershell you basically need to first load the dll eg
[void][Reflection.Assembly]::LoadFile("C:\temp\EWSUtil.dll")
Then you can use the objects that are defined within this class the first thing you need to do is create a EWS connection (this is a custom class I've created within the Class library that contains a Exchange Service Binding for EWS). The parameters you pass into the objects creation affect what authentication is used (eg impersonation or delegation) and also whether autodiscover is used or not. I haven't included any overloads so all the parameters are mandatory you just need to pass in $null if you don't want to use some. So the parameters for the EWSConnection object.
$casURL = "https://" + $servername + "/EWS/Exchange.asmx"
$ewc = new-object EWSUtil.EWSConnection("user@domamain.com",$false, "username", "password", "domain", $casURL )
- This is the email address of the mailbox you want to create the feed from.
- This is a boolean that indicates whether you want to use EWS impersonation to access the mailbox you specified in 1. If this is set to $false then delegate access is used.
- This is the username to use if you want to specify implicant credentials if you want to use the currently logged on user set this to $null
- This is the password to use if you want to specify implicant credentials if you want to use the currently logged on user set this to $null
- This is the domain to use if you want to specify implicant credentials if you want to use the currently logged on user set this to $null
- This is the URL for the CAS server to use if you set this to $null the library will try to use autodiscover to find a CAS server URL. (this isn't site aware)
The library returns a generic List of objects that matchs the restiction which depending on the code you use can be the messageID or to be as multi use as possible you can use any of the unindexed properties such as the Subject Etc.
The following is a sample that uses firstly the get-messagetrackinglogs cmdlet to retrieve the MessageID for any messages that are over 5 MB that have been recieved today in the last 2 hours. Then it looks at the recipient address and verfies that they are local accounts and then uses get-user to make sure you are working with the Primary Email address for the mailbox and not a proxy address which would make this script fail. It then uses Impersonation and the currently logged on user credentials to search for the message in the inbox of the recipient and exports the message and any attachments to c:\export\message and c:\export\attachments respectivly.
[void][Reflection.Assembly]::LoadFile("C:\temp\EWSUtil.dll")
function getMessage($recpAddress){
$recp
$ascii = new-object System.Text.ASCIIEncoding
$ewc = new-object EWSUtil.EWSConnection($recpAddress,$true, "", "", "",$null)
$dType = new-object EWSUtil.EWS.DistinguishedFolderIdType
$dType.Id = [EWSUtil.EWS.DistinguishedFolderIdNameType]::inbox
$fldarry = new-object EWSUtil.EWS.BaseFolderIdType[] 1
$fldarry[0] = $dType
$randNumber = New-Object system.random
$prop = new-object EWSUtil.EWS.PathToUnindexedFieldType
$prop.FieldURI = [EWSUtil.EWS.UnindexedFieldURIType]::messageInternetMessageId
$messagelist = $ewc.FindMessage($fldarry,$prop,$_.MessageID,$false)
foreach ($message in $messagelist){
$baByteArray = [Convert]::FromBase64String($message.MimeContent.Value)
$emlMessage = $ascii.GetString($baByteArray)
$emlfile = new-object IO.StreamWriter(("c:\export\message\" + $message.Subject.Replace("#","").Replace(":","") + $mc+ ".eml"),$true)
$emlfile.WriteLine($emlMessage)
$emlfile.Close()
"Exported Message " + $message.Subject
$mc = $mc +1
if ($message.hasattachments){
"Exported Message to " + ("c:\message\" + $message.Subject + $mc + ".eml")
foreach($attach in $message.Attachments){
$ewc.DownloadAttachment(("c:\export\Attachments\" + $randNumber.next(1,1000) + $attach.Name.ToString()),$attach.AttachmentId);
"Downloaded Attachment : " + $attach.Name.ToString()
}
}
}
}
$servername = "servername"
$DomainHash = @{ }
get-accepteddomain ForEach-Object{
if ($_.DomainType -eq "Authoritative"){
$DomainHash.add($_.DomainName.SmtpDomain.ToString().ToLower(),1)
}
}
$dtQueryDT = [DateTime]::UtcNow.AddHours(-2)
$dtQueryDTf = [DateTime]::UtcNow
Get-MessageTrackingLog -Server $servername -ResultSize Unlimited -Start $dtQueryDT -End $dtQueryDTf -EventId "RECEIVE" where {$_.TotalBytes -gt 5242880} ForEach-Object{
foreach($recp in $_.recipients){
if ($recp -ne ""){
$recparray = $recp.split("@")
if ($DomainHash.ContainsKey($recparray[1])){
$vuser = get-user $recp
getMessage($vuser.WindowsEmailAddress)
}
}
}}
To make this script search the whole mailbox instead of just the inbox you would need to change the following two lines,
first you need to set the parent folder to the root of the mailbox
$dType.Id = [EWSUtil.EWS.DistinguishedFolderIdNameType]::msgfolderroot
And change the following line to RecurseFolder
$messagelist = $ewc.RecurseFolder($fldarry,$prop,$_.MessageID,$false)
If you wanted to do a dumpsterdive change the last parameter to $true
I'm really just scratching the surface of the cool things you can do with this and i could keep rabbiting on for days but I've got bring this to end. I've put a download of the lastest verion of the library along with the source for those that want to roll your own here the sample script can be downloaded from here. As i mentioned before this library is untested and highly experimental and only sutible for test enviroments.