Wednesday, January 24, 2007

Exporting a Mailbox larger then 2 GB and spanning it across multiple PST files with a script

*update* I suggest checking out Michael Smith's RDO port of this script which is a much more developed examples http://theessentialexchange.com/blogs/michael/archive/2009/10/16/exporting-mailboxes-larger-than-2-gb-on-an-exchange-server.aspx . *update*

I have a small customer with an old server that recently had a staff member leave who had a very large mailbox over 5 GB. Now these guys are pretty tight still using Exchange 2000 and Office 2000 and don’t want to upgrade software or hardware so getting rid of this mailbox out the Exchange store while still giving people occasionally access to this mailbox was desirable and also challenging. Because all the clients are using Outlook 2000 I’m stuck with the 2 GB PST non-Unicode file limit which Exmerge is similarly afflicted with. Using Exmerge with date filters was one possible solution but I decided I’d rather just do a liner export where mail was exported one item at a time and would just span to the next PST when the space was exhausted in one. This did prove a little changeling at first but I did manage to come up with a script that worked using a combination of CDO 1.2 and RDO (Redemption Data Objects) which is part of Dmitry Streblechenko excellent Outlook Redemption library. RDO provides the ability to create PST files on the fly and also provides the same functionality as CDO 1.2. I did have some problems copying different types of objects with both libraries so I found using a combination of both allowed me to work around the problems I did have and successfully copy most objects from a mailbox.

How does it work?

The script is broken up into multiple subs and functions that all perform different tasks the first task that needs to be done is to logon to the mailbox and create the first PST file. Once that PST file is created the delete-items folder is located and a mapping is created using a dictionary object that maps the entryID of the mailboxe’s deleted items to the EntryID of the deleted items in the PST file. The script then loops through every folder in the Mailbox and then recreates this hierarchy in the PST file. This will happen every time the createpst function is called and ensures that the same folder structure is in each of the spanned PST files. EntryID mappings are created in the dictionary object for each folder as is used later on in the script to process each the contents of each folder and map the items that are being copied in to the right folder in the PST file. The CreatePST function uses the enumfolder function to enumerate any subfolders and the ProcessFolderRoot and ProcessFolderSub to create the folders in the PST and update the dictionary object.

Once the PST is setup the script then performs another enumeration on the folders in the mailbox and processes the items in each of the folder collections in the processitems sub. This sub loops though each item in a folder one at a time and calculates what the size of the PST file will be after copying the current item into the file based on the size of the item and the current file size of the PST file returned form the File System Object. If the new file size will be over a configure threshold value a new PST file is created by calling the CreatePST function. The destination folder object is reset and the script then resumes using the CDO 1.2 message object’s copyto function. If the cdo copyto function fails the RDO copyto function is tried as a fallback (this is a work around for some issues I had with both libraries). If the processitems sub detects the current folder is a contacts folder RDO is used to do the copy to stop the script from hitting the CDO security prompt issues. To track the progress of the script debugging information is written to the command prompt and also to a file in the same directory as the PST file. After each folder is processed the number of items in both the source and destination are written to the log file so this can be checked for debugging purposes.

How to use the script

The script requires CDO 1.2 (from Exchange, Outlook or the Standalone version) and Redemption. The script has a number of parameters you need to configure first


tnThreshold = 1800

This is the threshold value for the size of the spanned PST’s I found using 1800 to be the most effective threshold and made sure that the pst files where always under 2 GB. If you want to make the PST smaller eg maybe you want to fit the spanned PST's onto multiple CD’s you could reduce this value it doesn’t work out to be an exact size so you need to make sure you give yourself a buffer.

servername = "servername"

mbMailbox = "mailbox"


These variables should be obvious

bfBaseFilename = "expMailbox"

pfFilePath = "c:\temp\"


The base file name is what the exported PST will be called there will also be have a number appended to the file name to specify the number in the span set.


Note: This script is defiantly not a replacement for Exmerge or any other backup or export methods you have. (eg if I had Outlook 2003 at this client I would have used this instead). I can’t verify the accuracy or consistency of this script when exporting a mailbox so it’s use at your own risk and do your own testing. (Any always make sure you have backed up what ever you’re exporting on a different media). For me it seemed to work fine exporting my 5 GB mailbox to 3 pst file which can be accessed from the old outlook 2000 clients okay when ever needed.

I’ve put a downloadable copy of this script here the script itself looks like

Set doDictionaryObject = CreateObject("Scripting.Dictionary")
Set fso = CreateObject("Scripting.FileSystemObject")
Set fso = CreateObject("Scripting.FileSystemObject")
set RDOSession = CreateObject("Redemption.RDOSession")
tsize = 10
tnThreshold = 1800
servername = "ServerName"
mbMailbox = "Mailbox"
bfBaseFilename = "expMailbox"
pfFilePath = "c:\temp\"
fnFileName = ""
PST = ""
pstroot = ""
IPMRoot = ""
pfPstFile = ""
fNumber = 0
set wfile = fso.opentextfile(pfFilePath & bfBaseFilename & ".txt",2,true)
RDOSession.LogonExchangeMailbox mbMailbox,servername
Set dfDeletedItemsFolder = RDOSession.GetDefaultFolder(3)
CreateNewPst()

wscript.echo fnFileName
wscript.echo "Enumerate Messages"
for miLoop = 1 to IPMRoot.Folders.count
ProcessItems(IPMRoot.Folders(miLoop))
if IPMRoot.Folders(miLoop).Folders.count <> 0 then
call Enumfolders(IPMRoot.Folders(miLoop),PstRootFolder,2)
end if
next


function Enumfolders(FLDS,RootFolder,ltype)
for fl = 1 to FLDS.Folders.count
if ltype = 1 then
call ProcessFolderSub(FLDS.folders(fl),RootFolder)
else
ProcessItems(FLDS.folders(fl))
end if
wscript.echo FLDS.folders(fl).Name
if FLDS.folders(fl).Folders.count <> 0 then
if ltype = 1 then
call Enumfolders(FLDS.folders(fl),FLDS.folders(fl).EntryID,1)
else
call Enumfolders(FLDS.folders(fl),FLDS.folders(fl).EntryID,2)
end if
end if
next
End function

Function CreateNewPst()

doDictionaryObject.RemoveAll
fNumber = fNumber + 1
fnFileName = pfFilePath & bfBaseFilename & "-" & fNumber & ".pst"
set PST = RDOSession.Stores.AddPSTStore(fnFileName, 1, "Exported MailBox-" & now())
set pstroot = RDOSession.GetFolderFromID(PST.IPMRootFolder.EntryID,PST.EntryID)
For Each pstfld In PstRoot.folders
If pstfld.Name = "Deleted Items" Then
If fNumber = 1 Then
doDictionaryObject.add dfDeletedItemsFolder.EntryID, pstfld.EntryID
wscript.echo "Added Deleted Items Folder"
End if
End if
next
set IPMRoot = RDOSession.Stores.DefaultStore.IPMRootFolder
for fiLoop = 1 to IPMRoot.Folders.count
if IPMRoot.Folders(fiLoop).Name <> "Deleted Items" then
PstRootFolder = ProcessFolderRoot(IPMRoot.Folders(fiLoop),PST.IPMRootFolder.EntryID)
if IPMRoot.Folders(fiLoop).Folders.count <> 0 then
call Enumfolders(IPMRoot.Folders(fiLoop),IPMRoot.Folders(fiLoop).EntryID,1)
end If
Else
if IPMRoot.Folders(fiLoop).Folders.count <> 0 then
call Enumfolders(IPMRoot.Folders(fiLoop),IPMRoot.Folders(fiLoop).EntryID,1)
end if
end if
next
Set pfPstFile = fso.GetFile(fnFileName)

end function

function ProcessFolderRoot(Fld,parentfld)

set CDOPstfld = RDOSession.GetFolderFromID(parentfld,PST.EntryID)
wscript.echo fld.Name
Set newFolder = CDOPstfld.Folders.ADD(Fld.Name)
ProcessFolder = newfolder.EntryID
newfolder.fields(&H3613001E) = Fld.fields(&H3613001E)

doDictionaryObject.add Fld.EntryID,newfolder.EntryID
end Function

function ProcessFolderSub(Fld,parentfld)

set CDOPstfld = RDOSession.GetFolderFromID(doDictionaryObject.item(parentfld),PST.EntryID)
wscript.echo fld.Name
Set newFolder = CDOPstfld.Folders.ADD(Fld.Name)
ProcessFolder = newfolder.EntryID
newfolder.fields(&H3613001E) = Fld.fields(&H3613001E)

doDictionaryObject.add Fld.EntryID,newfolder.EntryID
end function

Sub ProcessItems(Fld)

If Fld.fields(&H3613001E) = "IPF.Contact" Then
set dfDestinationFolder = RDOSession.GetFolderFromID(doDictionaryObject.item(Fld.EntryID),PST.EntryID)
wscript.echo dfDestinationFolder.Name
wfile.writeLine("Processing Folder : ") & dfDestinationFolder.Name
for fiItemloop = 1 to Fld.items.count
on error resume next
pfPredictednewSize = formatnumber((pfPstFile.size + Fld.items(fiItemloop).size)/1048576,2,0,0,0)
if err.number <> 0 Then
Wscript.echo "Error Processing Item in " & Fld.Name
wscript.echo "EntryID of Item:"
wscript.echo Fld.items(fiItemloop).EntryID
wscript.echo "Subect of Item:"
wscript.echo Fld.items(fiItemloop).Subject
Wfile.writeline("Error Processing Item in " & Fld.Name)
Wfile.writeline("EntryID of Item:")
Wfile.writeline(Fld.items(fiItemloop).EntryID )
Wfile.writeline("Subect of Item:")
Wfile.writeline(Fld.items(fiItemloop).Subject)
err.clear
end if
If Int(pfPredictednewSize) >= Int(tsize) Then
Wscript.echo "10 MB Exported"
tsize = tsize + 10
End if
If Int(pfPredictednewSize) >= Int(tnThreshold) Then
wfile.writeLine("New PST about to be created - Destination - Number of Items : " & dfDestinationFolder.messages.count)
CreateNewPst()
set dfDestinationFolder = RDOSession.GetFolderFromID(doDictionaryObject.item(Fld.EntryID),PST.EntryID)
call Fld.items(fiItemloop).copyto(dfDestinationFolder)
if err.number <> 0 then
Wscript.echo "Error Processing Item in " & Fld.Name
wscript.echo "EntryID of Item:"
wscript.echo Fld.items(fiItemloop).EntryID
wscript.echo "Subect of Item:"
wscript.echo Fld.items(fiItemloop).Subject
Wfile.writeline("Error Processing Item in " & Fld.Name)
Wfile.writeline("EntryID of Item:")
Wfile.writeline(Fld.items(fiItemloop).EntryID )
Wfile.writeline("Subect of Item:")
Wfile.writeline(Fld.items(fiItemloop).Subject)
err.clear
end if
else
call Fld.items(fiItemloop).copyto(dfDestinationFolder)
if err.number <> 0 then
Wscript.echo "Error Processing Item in " & Fld.Name
wscript.echo "EntryID of Item:"
wscript.echo Fld.items(fiItemloop).EntryID
wscript.echo "Subect of Item:"
wscript.echo Fld.items(fiItemloop).Subject
Wfile.writeline("Error Processing Item in " & Fld.Name)
Wfile.writeline("EntryID of Item:")
Wfile.writeline(Fld.items(fiItemloop).EntryID )
Wfile.writeline("Subect of Item:")
Wfile.writeline(Fld.items(fiItemloop).Subject)
err.clear
end if
End if
on error goto 0
Next
wfile.writeLine("Source - Number of Items : " & Fld.fields(&h36020003) & " Destination - Number of Items : " & dfDestinationFolder.items.count)
else
set CDOSession = CreateObject("MAPI.Session")
CDOSession.MAPIOBJECT = RDOSession.MAPIOBJECT
Set objInfoStore = CDOSession.GetInfoStore(PST.EntryID)
set srcFld = CDOSession.GetFolder(Fld.EntryID)
wfile.writeLine("Processing Folder : ") & srcFld.Name
set dfDestinationFolder = CDOSession.GetFolder(doDictionaryObject.item(Fld.EntryID),PST.EntryID)
wscript.echo dfDestinationFolder.Name
for fiItemloop = 1 to srcFld.messages.count
on error resume next
pfPredictednewSize = formatnumber((pfPstFile.size + srcFld.messages(fiItemloop).size)/1048576,2,0,0,0)
if err.number <> 0 Then
Wscript.echo "Error Processing Item in " & srcFld.messages(fiItemloop).Name
wscript.echo "EntryID of Item:"
wscript.echo srcFld.messages(fiItemloop).id
wscript.echo "Subect of Item:"
wscript.echo srcFld.messages(fiItemloop).Subject
Wfile.writeline("Error Processing Item in " & srcFld.messages(fiItemloop).Name)
Wfile.writeline("EntryID of Item:")
Wfile.writeline(srcFld.messages(fiItemloop).id)
Wfile.writeline("Subect of Item:")
Wfile.writeline(srcFld.messages(fiItemloop).Subject)
err.clear
rem Try to Copy with RDO
Set rdosrc = RDOSession.GetMessageFromID(srcFld.messages(fiItemloop).Id)
rdosrc.copyto(dfDestinationFolder)
if err.number <> 0 Then
Wscript.echo "Also Failed RDO Copy"
wfile.writeline("Also Failed RDO Copy")
Else
Wscript.echo "Copied with RDO Okay"
wfile.writeline("Copied with RDO Okay")
End if
err.clear
end if
If Int(pfPredictednewSize) >= Int(tsize) Then
Wscript.echo "10 MB Exported"
tsize = tsize + 10
End if
If Int(pfPredictednewSize) >= Int(tnThreshold) Then
wfile.writeLine("New PST about to be created - Destination - Number of Items : " & dfDestinationFolder.messages.count)
CreateNewPst()
set CDOSession = CreateObject("MAPI.Session")
CDOSession.MAPIOBJECT = RDOSession.MAPIOBJECT
Set objInfoStore = CDOSession.GetInfoStore(PST.EntryID)
set dfDestinationFolder = CDOSession.GetFolder(doDictionaryObject.item(Fld.EntryID),PST.EntryID)
Set cpymsg = srcFld.messages(fiItemloop).copyto(dfDestinationFolder.ID)
cpymsg.update
if err.number <> 0 then
Wscript.echo "Error Processing Item in " & Fld.Name
wscript.echo "EntryID of Item:"
wscript.echo srcFld.messages(fiItemloop).Id
wscript.echo "Subect of Item:"
wscript.echo srcFld.messages(fiItemloop).Subject
Wfile.writeline("Error Processing Item in " & Fld.Name)
Wfile.writeline("EntryID of Item:")
Wfile.writeline(srcFld.messages(fiItemloop).id)
Wfile.writeline("Subect of Item:")
Wfile.writeline(srcFld.messages(fiItemloop).Subject)
err.clear
rem Try to Copy with RDO
Set rdosrc = RDOSession.GetMessageFromID(srcFld.messages(fiItemloop).Id)
rdosrc.copyto(dfDestinationFolder)
if err.number <> 0 Then
Wscript.echo "Also Failed RDO Copy"
wfile.writeline("Also Failed RDO Copy")
Else
Wscript.echo "Copied with RDO Okay"
wfile.writeline("Copied with RDO Okay")
End if
err.clear
end if
Else
Set cpymsg = srcFld.messages(fiItemloop).copyto(dfDestinationFolder.ID)
cpymsg.update
if err.number <> 0 then
Wscript.echo "Error Processing Item in " & Fld.Name
wscript.echo "EntryID of Item:"
wscript.echo srcFld.messages(fiItemloop).id
wscript.echo "Subect of Item:"
wscript.echo srcFld.messages(fiItemloop).Subject
Wfile.writeline("Error Processing Item in " & Fld.Name)
Wfile.writeline("EntryID of Item:")
Wfile.writeline(srcFld.messages(fiItemloop).id)
Wfile.writeline("Subect of Item:")
Wfile.writeline(srcFld.messages(fiItemloop).Subject)
err.clear
rem Try to Copy with RDO
Set rdosrc = RDOSession.GetMessageFromID(srcFld.messages(fiItemloop).Id)
rdosrc.copyto(dfDestinationFolder)
if err.number <> 0 Then
Wscript.echo "Also Failed RDO Copy"
wfile.writeline("Also Failed RDO Copy")
Else
Wscript.echo "Copied with RDO Okay"
wfile.writeline("Copied with RDO Okay")
End if
err.clear
end if
End if
on error goto 0
Next
wfile.writeLine("Source - Number of Items : " & srcFld.fields(&h36020003) & " Destination - Number of Items : " & dfDestinationFolder.messages.count)
End if
end sub

26 comments:

Chris said...

awsome script! very handy. Doesn't seem to like some emails but for the most part saved me alot of time.

Michael said...

This is awesome!!!
thank you so much for posting this!

Anonymous said...

Glenn, how is it that you can do what MS can't? I've been waiting for ages for Exmerge to support mailboxes > 2GB.

Glen said...

Exchange 2007's Export mailbox can do this now. As i said in this post make sure you keep an eye on the error logs and do your own testing to ensure you don't loose any data.

Cheers
Glen

Anonymous said...

Great script...thank you!

Just glancing at the code, you've created a simple solution to a lingering problem in the M$ world.

I may possibly expand on your script to allow for sorting of items based on date fields for my use...I would, of course, credit you for the original script.

If I do, I'll post back.

Thanks again!

Anonymous said...

This sound like what I was looking for.

However i've got some issue on maybe you could help me.

I've installed the CDO from Microsoft then Redemption DLL.

After that I run the script and it fail on line 4 "set RDOSession = CreateObject("Redemption.RDOSession")"

It's strange cause the DLL seems to be registered as well...

Any idea? Thanks anyway for the great job you've done so far.

Anonymous said...

Oups nevermind, I didn't noticed Iwas running it on x64 server... You can delete these post!

Darrell Porter said...

Hi Glen,
I have not yet had an opportunity to review the script in-depth. Is it limited to mail, or does it also handle calendars, tasks, contacts, etc?

Glen said...

Yes it does all items

Cheers
Glen

bradzilla said...

Hi Glen,
Many thanks for this site - it rocks!! I'm getting an error when I run the spanning script - "This key is already associated with an element of this collection", line 90, char 1. The code is 800A01C9. I'm fairly sure this is because there are duplicate folder names in the mailbox. Anyway to work around this in the script?

WalkaboutTigger said...

Is there a way to limit the script by date? For example, is there a way to export all items prior to August 1, 2009 at 12:01 am?

Thanks!

WalkaboutTigger said...

And do you have any idea why an export using the script would be incredibly slow? I am getting less than 300 MB an hour.

Thanks!

Glen said...

The foldername isn't used in the Hashtable the Folder EntryID is used maybe whats happening is the folder EntryID is null or 0 you could try to echo out the value for the folder ID before this line runs then you will know if its an issue with a blank or null folder id.

eg

wscript.echo Fld.EntryID

Cheers
Glen

Glen said...

To limit the script by date you would need to use filters. So it would require moding the ProcessItems sub routing and using something like eg using the Find Method in the Item collection http://www.dimastr.com/redemption/rdo/RDOItems.htm.

Why is it slow well it processing every item in the mailbox probably an I/O issue either on you Exchange server or workstation where you running the script. Eg check things like the processor utilization on the workstation etc. Also virus checker can slow things down if they are scanning on every update of PST etc.

Cheers
Glen

Netizen said...

This has just saved me a big headache, thank you muchly!

One tweak I had to make was to stop it prompting me every 10MB it copied, which was rather annoying on a 4GB mailbox. I changed the tsize value along with tsize = tsize + 100 to get a prompt every 100MB.

Pieter Thoma said...

Thanks for the script. Great work!

Marc said...

I'm trying this script but only runs properly when I export the administrator mailbox, with any other mailbox I receive next error:

Redemption.RDOFolders: Error in IMAPIFolder.Create: MAPI_E_COLLISION
ulversion:0
Error: Folder already exists (???)
Component: Personal Folders
ulLowLevelError:0
ulContext:805437697

With any user with administrator rights can export the administrator mailbox but no any other mailbox. Ex: admin user (with administrator rights) can export administrator mailbox but cannot "admin mailbox".
Any idea about what's happening?

nickname said...

I was looking for the exact same solution without paying $$$$ to unverified packages..And very glad to find your priceless codes and explanations..I will try to use to extract old and big size mailboxes to pst file, and see if it work..

Thanks million!!

chrisl said...

Glen, the script exports 10MB at a time, why? I have to press ok with the WSH script message for it to continue. Any ideas?

Anonymous said...

Got the same error message (folder duplication) as Marc.

One solution is to concatenate the folder names with random numbers or with counter etc etc. Right now the script is useless for me. ):
>>>
Set newFolder = CDOPstfld.Folders.ADD(Fld.Name)
(for root and for sub folders)

i dont program in vb so i dont want to mess up the code.

Thank You,

Anonymous said...

Also i should click Ok on every mail folder the script is creating, why? Thanks.

Anonymous said...

Hi,

I get the following error,

Line: 17
Char: 1
Error: Path not found
Code: 800A004C

Anyone have any idea what this might be?

Thanks in advance
Meyer

Glen Scales said...

It use c:\temp by default if this folder doesn't exist then you will get an error

Anonymous said...

Awesome, thanks.

Now I'm getting the following

Line: 18
char: 1
Error: Error in IMsgServiceAdmin::ConfigureMsgService: MAIL_E_NAMENOTFOUND (or WSAECONNRESET)
Code: 81002746

Thanks
Meyer

Glen Scales said...

That means its failing to resolve the Name of the Mailbox you trying to export you can have a read of http://blogs.msdn.com/b/stephen_griffin/archive/2011/10/13/the-elusive-0x81002746-error.aspx to see if any of these apply (eg if you have hidden the Mailbox from the GAL etc or don't have rights).

Cheers
Glen

Havoc said...

It is a bug, you probably have a non-english version in which case the Deleted Items-folder is not called just that,

Just replace all references to "Deleted Items" with the variable dfDeletedItemsFolder and it will work again. Same with the improved RDO-script linked at the top.