One of all the more useful reasons to learn scripting is the ability to turn information that is recorded in one format in a seemless useless rable of bits, into a format this is more useful to ordinary humans in everyday situations. One the examples of this are the IIS logs which in Exchange contain information about users accessing OWA, Accessing ActiveSync and Outlook Anywhere. Like everything in IT there are a few ways of tackling how you go about turing log information into something useful, one of the more popular ways to do this is using the Log parser which is a brilliant tool for those that aren't comfortable doing a lot of coding. You cant beat this tool for speed and efficiency and if your parsing logs often you should learn to use it. More recently the exLogAnalyser has been released which is interesting and it looks like a really great piece of coding that lacks a very very important ingredients for a tool like this. The documentation is spartan and it was written to solve a problem rather then help other people solve their own problems. (It has great potential you just need think about how people will use it!).
If you want to do something a lot more creative with the information your parsing out of log files this is when you need to do some serious number crunching and processing. The first really big problem when parsing a log file is that these files can get really large and reading a very large raw log file that contains millions of lines can be very slow if you want to process the information in everyline in a specific way. If done poorly it can mean your script will start consuming a lot of memory or maybe just take a lot of time. This is where using LogParser generally has a large advantage but the cost is what you can creatively do with the result so the trade off with using Powershell for very large log files is you get a lot of functionality but you pay a big cost in performance when you do it. The good thing however is that not everybody has that problem of very large log files (its really only if you have a large number of users) so for those that don't or dont mind spending some CPU cycles on processing then I've got a few scripts that you might find usefull. The parse engine is built on a previous post which works well in parsing the log file in a more reliable way. One goal I had for this script was to create something that would track logons so it would show me the logon/logoff time the duration and how many time a users logs on. To do this the script uses a hashtable and looks for the contiuned appearance of a certain entry from a user in a log file. If there is a gap of more then 30 mintues between the last entry it track this as a logon and if the users reappears in the sequence this becomes a new logon.
The script shows summaries or raw information and does time conversation to local time as well. I've posted a download of this script here the script iteself looks like the following.
[System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")
[System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms")
function openLog{
$exFileName = new-object System.Windows.Forms.openFileDialog
$exFileName.ShowHelp = $true
$exFileName.ShowDialog()
$fnFileNamelableBox.Text = $exFileName.FileName
Populatetable
}
#
function Populatetable{
$logTable.Clear()
$sumTable.Clear()
$fname = $fnFileNamelableBox.Text
$mbcombCollection = @()
$FldHash = @{}
$usHash = @{}
$sumHash = @{}
$fieldsline = (Get-Content $fname)[3]
$fldarray = $fieldsline.Split(" ")
$fnum = -1
foreach ($fld in $fldarray){
$FldHash.add($fld,$fnum)
$fnum++
}
get-content $fname | Where-Object -FilterScript { $_ -ilike “*MSRPC*” } | %{
$lnum ++
if ($lnum -eq $rnma){ Write-Progress -Activity "Read Lines" -Status $lnum
$rnma = $rnma + 1000
}
$linarr = $_.split(" ")
$uid = $linarr[$FldHash["cs-username"]] + $linarr[$FldHash["c-ip"]]
if ($linarr[$FldHash["cs-username"]].length -gt 2){
if ($usHash.Containskey($uid) -eq $false){
if ($linarr.Length -gt 0){$lfDate = $linarr[$FldHash["date"]]}
if ($linarr.Length -gt 1){$lfTime = $linarr[$FldHash["time"]]}
$ldLogDatecombine = $lfDate + " " + $lfTime
if ($ltimeconv.Checked -eq $true){
$LogDate = Get-Date($ldLogDatecombine)
$LocalLogDate = $LogDate.tolocaltime()
$ldLogDatecombine = $LocalLogDate.ToString("yyyy-MM-dd HH:mm:ss")
}
$usrobj = "" | select UserName,IpAddress,LogonTime,LogOffTime,Duration,NumberofRequests,BytesSent,BytesRecieved
$usrobj.UserName = $linarr[$FldHash["cs-username"]]
$usrobj.IpAddress = $linarr[$FldHash["c-ip"]]
$usrobj.LogonTime = $ldLogDatecombine
$usrobj.LogOffTime = $ldLogDatecombine
$usrobj.Duration = 0
$usrobj.NumberofRequests = 0
$usrobj.BytesSent = $linarr[$FldHash["sc-bytes"]]
$usrobj.BytesRecieved = $linarr[$FldHash["cs-bytes"]]
$usHash.add($uid,$usrobj)
}
else{
if ($linarr.Length -gt 0){$lfDate = $linarr[$FldHash["date"]]}
if ($linarr.Length -gt 1){$lfTime = $linarr[$FldHash["time"]]}
$ldLogDatecombine = $lfDate + " " + $lfTime
if ($ltimeconv.Checked -eq $true){
$LogDate = Get-Date($ldLogDatecombine)
$LocalLogDate = $LogDate.tolocaltime()
$ldLogDatecombine = $LocalLogDate.ToString("yyyy-MM-dd HH:mm:ss")
}
$duration = New-TimeSpan $usHash[$uid].LogOffTime $ldLogDatecombine
if ([INT]$duration.Totalminutes -gt 30){
$mbcombCollection += $usHash[$uid]
$usHash.remove($uid)
$usrobj = "" | select UserName,IpAddress,LogonTime,LogOffTime,Duration,NumberofRequests,BytesSent,BytesRecieved
$usrobj.UserName = $linarr[$FldHash["cs-username"]]
$usrobj.IpAddress = $linarr[$FldHash["c-ip"]]
$usrobj.LogonTime = $ldLogDatecombine
$usrobj.LogOffTime = $ldLogDatecombine
$usrobj.BytesSent = $linarr[$FldHash["sc-bytes"]]
$usrobj.BytesRecieved = $linarr[$FldHash["cs-bytes"]]
$usrobj.Duration = 0
$usrobj.NumberofRequests = 0
$usHash.add($uid,$usrobj)
}
else{
$usHash[$uid].LogOffTime = $ldLogDatecombine
$lgduration = New-TimeSpan $usHash[$uid].LogonTime $ldLogDatecombine
$usHash[$uid].Duration = [INT]$lgduration.Totalminutes
$usrobj.NumberofRequests = [INT]$usrobj.NumberofRequests + 1
$usrobj.BytesSent = [INT]$usrobj.BytesSent + [INT]$linarr[$FldHash["sc-bytes"]]
$usrobj.BytesRecieved = [INT]$usrobj.BytesRecieved + [INT]$linarr[$FldHash["cs-bytes"]]
}
}
}
}
$usHash.GetEnumerator() | sort LogonTime -descending | foreach-object {
$logTable.Rows.Add($_.value.UserName,$_.value.IpAddress,$_.value.LogonTime,$_.value.LogOffTime,$_.value.Duration,$_.value.NumberofRequests,$_.value.BytesSent,$_.value.BytesRecieved)
if($sumHash.contains($_.value.UserName)){
$sumHash[$_.value.UserName].NumberofLogons = $sumHash[$_.value.UserName].NumberofLogons + 1
$sumHash[$_.value.UserName].Duration = [INT]$sumHash[$_.value.UserName].Duration + [INT]$_.value.Duration
$sumHash[$_.value.UserName].BytesSent = [INT]$sumHash[$_.value.UserName].BytesSent + [INT]$_.value.BytesSent
$sumHash[$_.value.UserName].BytesRecieved = [INT]$sumHash[$_.value.UserName].BytesRecieved + [INT]$_.value.BytesRecieved
}
else{
$usrobj = "" | select UserName,NumberofLogons,Duration,BytesSent,BytesRecieved
$usrobj.UserName = $_.value.UserName
$usrobj.NumberofLogons = 1
$usrobj.Duration = $_.value.Duration
$usrobj.BytesSent = $_.value.BytesSent
$usrobj.BytesRecieved = $_.value.BytesRecieved
$sumHash.add($_.value.UserName,$usrobj)
}
}
$sumHash.GetEnumerator() | sort LogonTime -descending | foreach-object {
$sumTable.Rows.Add($_.value.UserName,$_.value.NumberofLogons,$_.value.Duration,$_.value.BytesSent,$_.value.BytesRecieved)
}
$dgDataGrid.DataSource = $sumTable
}
$Dataset = New-Object System.Data.DataSet
$logTable = New-Object System.Data.DataTable
$logTable.TableName = "RCPLogons"
$logTable.Columns.Add("UserName");
$logTable.Columns.Add("IpAddress");
$logTable.Columns.Add("LogonTime");
$logTable.Columns.Add("LogOffTime");
$logTable.Columns.Add("Duration",[INT64]);
$logTable.Columns.Add("NumberofRequests",[INT64]);
$logTable.Columns.Add("Sent",[INT64]);
$logTable.Columns.Add("Recieved",[INT64]);
$sumTable = New-Object System.Data.DataTable
$sumTable.TableName = "RpcSummary"
$sumTable.Columns.Add("UserName");
$sumTable.Columns.Add("NumberofLogons",[INT64]);
$sumTable.Columns.Add("Duration",[INT64]);
$sumTable.Columns.Add("Sent",[INT64]);
$sumTable.Columns.Add("Recieved",[INT64]);
$form = new-object System.Windows.Forms.form
$form.Text = "Outlook Anywhere Log Tool"
# Add DataGrid View
# Local Time Converstion CheckBox
$ltimeconv = new-object System.Windows.Forms.CheckBox
$ltimeconv.Location = new-object System.Drawing.Size(310,7)
$ltimeconv.Size = new-object System.Drawing.Size(150,20)
$ltimeconv.Checked = $true
$ltimeconv.Text = "Convert to Local Time"
$form.Controls.Add($ltimeconv)
# Content
$cmClickMenu = new-object System.Windows.Forms.ContextMenuStrip
$cmClickMenu.Items.add("test122")
# Add Open Log file Button
$olButton = new-object System.Windows.Forms.Button
$olButton.Location = new-object System.Drawing.Size(20,19)
$olButton.Size = new-object System.Drawing.Size(75,23)
$olButton.Text = "Select file"
$olButton.Add_Click({openLog})
$form.Controls.Add($olButton)
# Add FileName Lable
$fnFileNamelableBox = new-object System.Windows.Forms.Label
$fnFileNamelableBox.Location = new-object System.Drawing.Size(110,25)
$fnFileNamelableBox.forecolor = "MenuHighlight"
$fnFileNamelableBox.size = new-object System.Drawing.Size(200,20)
$form.Controls.Add($fnFileNamelableBox)
# Add Refresh Log file Button
$refreshButton = new-object System.Windows.Forms.Button
$refreshButton.Location = new-object System.Drawing.Size(390,29)
$refreshButton.Size = new-object System.Drawing.Size(75,23)
$refreshButton.Text = "Refresh"
$refreshButton.Add_Click({Populatetable})
$form.Controls.Add($refreshButton)
# Show Summary CheckBox
$SsumBox = new-object System.Windows.Forms.CheckBox
$SsumBox.Location = new-object System.Drawing.Size(510,7)
$SsumBox.Size = new-object System.Drawing.Size(200,20)
$SsumBox.Checked = $true
$SsumBox.Add_Click({if ($SsumBox.Checked -eq $true){$dgDataGrid.DataSource = $sumTable}
else {$dgDataGrid.DataSource = $logTable}})
$SsumBox.Text = "Show Summary"
$form.Controls.Add($SsumBox)
$dgDataGrid = new-object System.windows.forms.DataGrid
$dgDataGrid.AllowSorting = $True
$dgDataGrid.Location = new-object System.Drawing.Size(12,81)
$dgDataGrid.size = new-object System.Drawing.Size(1024,750)
$form.Controls.Add($dgDataGrid)
$form.topmost = $true
$form.Add_Shown({$form.Activate()})
$form.ShowDialog()
If you want to do something a lot more creative with the information your parsing out of log files this is when you need to do some serious number crunching and processing. The first really big problem when parsing a log file is that these files can get really large and reading a very large raw log file that contains millions of lines can be very slow if you want to process the information in everyline in a specific way. If done poorly it can mean your script will start consuming a lot of memory or maybe just take a lot of time. This is where using LogParser generally has a large advantage but the cost is what you can creatively do with the result so the trade off with using Powershell for very large log files is you get a lot of functionality but you pay a big cost in performance when you do it. The good thing however is that not everybody has that problem of very large log files (its really only if you have a large number of users) so for those that don't or dont mind spending some CPU cycles on processing then I've got a few scripts that you might find usefull. The parse engine is built on a previous post which works well in parsing the log file in a more reliable way. One goal I had for this script was to create something that would track logons so it would show me the logon/logoff time the duration and how many time a users logs on. To do this the script uses a hashtable and looks for the contiuned appearance of a certain entry from a user in a log file. If there is a gap of more then 30 mintues between the last entry it track this as a logon and if the users reappears in the sequence this becomes a new logon.
The script shows summaries or raw information and does time conversation to local time as well. I've posted a download of this script here the script iteself looks like the following.
[System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")
[System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms")
function openLog{
$exFileName = new-object System.Windows.Forms.openFileDialog
$exFileName.ShowHelp = $true
$exFileName.ShowDialog()
$fnFileNamelableBox.Text = $exFileName.FileName
Populatetable
}
#
function Populatetable{
$logTable.Clear()
$sumTable.Clear()
$fname = $fnFileNamelableBox.Text
$mbcombCollection = @()
$FldHash = @{}
$usHash = @{}
$sumHash = @{}
$fieldsline = (Get-Content $fname)[3]
$fldarray = $fieldsline.Split(" ")
$fnum = -1
foreach ($fld in $fldarray){
$FldHash.add($fld,$fnum)
$fnum++
}
get-content $fname | Where-Object -FilterScript { $_ -ilike “*MSRPC*” } | %{
$lnum ++
if ($lnum -eq $rnma){ Write-Progress -Activity "Read Lines" -Status $lnum
$rnma = $rnma + 1000
}
$linarr = $_.split(" ")
$uid = $linarr[$FldHash["cs-username"]] + $linarr[$FldHash["c-ip"]]
if ($linarr[$FldHash["cs-username"]].length -gt 2){
if ($usHash.Containskey($uid) -eq $false){
if ($linarr.Length -gt 0){$lfDate = $linarr[$FldHash["date"]]}
if ($linarr.Length -gt 1){$lfTime = $linarr[$FldHash["time"]]}
$ldLogDatecombine = $lfDate + " " + $lfTime
if ($ltimeconv.Checked -eq $true){
$LogDate = Get-Date($ldLogDatecombine)
$LocalLogDate = $LogDate.tolocaltime()
$ldLogDatecombine = $LocalLogDate.ToString("yyyy-MM-dd HH:mm:ss")
}
$usrobj = "" | select UserName,IpAddress,LogonTime,LogOffTime,Duration,NumberofRequests,BytesSent,BytesRecieved
$usrobj.UserName = $linarr[$FldHash["cs-username"]]
$usrobj.IpAddress = $linarr[$FldHash["c-ip"]]
$usrobj.LogonTime = $ldLogDatecombine
$usrobj.LogOffTime = $ldLogDatecombine
$usrobj.Duration = 0
$usrobj.NumberofRequests = 0
$usrobj.BytesSent = $linarr[$FldHash["sc-bytes"]]
$usrobj.BytesRecieved = $linarr[$FldHash["cs-bytes"]]
$usHash.add($uid,$usrobj)
}
else{
if ($linarr.Length -gt 0){$lfDate = $linarr[$FldHash["date"]]}
if ($linarr.Length -gt 1){$lfTime = $linarr[$FldHash["time"]]}
$ldLogDatecombine = $lfDate + " " + $lfTime
if ($ltimeconv.Checked -eq $true){
$LogDate = Get-Date($ldLogDatecombine)
$LocalLogDate = $LogDate.tolocaltime()
$ldLogDatecombine = $LocalLogDate.ToString("yyyy-MM-dd HH:mm:ss")
}
$duration = New-TimeSpan $usHash[$uid].LogOffTime $ldLogDatecombine
if ([INT]$duration.Totalminutes -gt 30){
$mbcombCollection += $usHash[$uid]
$usHash.remove($uid)
$usrobj = "" | select UserName,IpAddress,LogonTime,LogOffTime,Duration,NumberofRequests,BytesSent,BytesRecieved
$usrobj.UserName = $linarr[$FldHash["cs-username"]]
$usrobj.IpAddress = $linarr[$FldHash["c-ip"]]
$usrobj.LogonTime = $ldLogDatecombine
$usrobj.LogOffTime = $ldLogDatecombine
$usrobj.BytesSent = $linarr[$FldHash["sc-bytes"]]
$usrobj.BytesRecieved = $linarr[$FldHash["cs-bytes"]]
$usrobj.Duration = 0
$usrobj.NumberofRequests = 0
$usHash.add($uid,$usrobj)
}
else{
$usHash[$uid].LogOffTime = $ldLogDatecombine
$lgduration = New-TimeSpan $usHash[$uid].LogonTime $ldLogDatecombine
$usHash[$uid].Duration = [INT]$lgduration.Totalminutes
$usrobj.NumberofRequests = [INT]$usrobj.NumberofRequests + 1
$usrobj.BytesSent = [INT]$usrobj.BytesSent + [INT]$linarr[$FldHash["sc-bytes"]]
$usrobj.BytesRecieved = [INT]$usrobj.BytesRecieved + [INT]$linarr[$FldHash["cs-bytes"]]
}
}
}
}
$usHash.GetEnumerator() | sort LogonTime -descending | foreach-object {
$logTable.Rows.Add($_.value.UserName,$_.value.IpAddress,$_.value.LogonTime,$_.value.LogOffTime,$_.value.Duration,$_.value.NumberofRequests,$_.value.BytesSent,$_.value.BytesRecieved)
if($sumHash.contains($_.value.UserName)){
$sumHash[$_.value.UserName].NumberofLogons = $sumHash[$_.value.UserName].NumberofLogons + 1
$sumHash[$_.value.UserName].Duration = [INT]$sumHash[$_.value.UserName].Duration + [INT]$_.value.Duration
$sumHash[$_.value.UserName].BytesSent = [INT]$sumHash[$_.value.UserName].BytesSent + [INT]$_.value.BytesSent
$sumHash[$_.value.UserName].BytesRecieved = [INT]$sumHash[$_.value.UserName].BytesRecieved + [INT]$_.value.BytesRecieved
}
else{
$usrobj = "" | select UserName,NumberofLogons,Duration,BytesSent,BytesRecieved
$usrobj.UserName = $_.value.UserName
$usrobj.NumberofLogons = 1
$usrobj.Duration = $_.value.Duration
$usrobj.BytesSent = $_.value.BytesSent
$usrobj.BytesRecieved = $_.value.BytesRecieved
$sumHash.add($_.value.UserName,$usrobj)
}
}
$sumHash.GetEnumerator() | sort LogonTime -descending | foreach-object {
$sumTable.Rows.Add($_.value.UserName,$_.value.NumberofLogons,$_.value.Duration,$_.value.BytesSent,$_.value.BytesRecieved)
}
$dgDataGrid.DataSource = $sumTable
}
$Dataset = New-Object System.Data.DataSet
$logTable = New-Object System.Data.DataTable
$logTable.TableName = "RCPLogons"
$logTable.Columns.Add("UserName");
$logTable.Columns.Add("IpAddress");
$logTable.Columns.Add("LogonTime");
$logTable.Columns.Add("LogOffTime");
$logTable.Columns.Add("Duration",[INT64]);
$logTable.Columns.Add("NumberofRequests",[INT64]);
$logTable.Columns.Add("Sent",[INT64]);
$logTable.Columns.Add("Recieved",[INT64]);
$sumTable = New-Object System.Data.DataTable
$sumTable.TableName = "RpcSummary"
$sumTable.Columns.Add("UserName");
$sumTable.Columns.Add("NumberofLogons",[INT64]);
$sumTable.Columns.Add("Duration",[INT64]);
$sumTable.Columns.Add("Sent",[INT64]);
$sumTable.Columns.Add("Recieved",[INT64]);
$form = new-object System.Windows.Forms.form
$form.Text = "Outlook Anywhere Log Tool"
# Add DataGrid View
# Local Time Converstion CheckBox
$ltimeconv = new-object System.Windows.Forms.CheckBox
$ltimeconv.Location = new-object System.Drawing.Size(310,7)
$ltimeconv.Size = new-object System.Drawing.Size(150,20)
$ltimeconv.Checked = $true
$ltimeconv.Text = "Convert to Local Time"
$form.Controls.Add($ltimeconv)
# Content
$cmClickMenu = new-object System.Windows.Forms.ContextMenuStrip
$cmClickMenu.Items.add("test122")
# Add Open Log file Button
$olButton = new-object System.Windows.Forms.Button
$olButton.Location = new-object System.Drawing.Size(20,19)
$olButton.Size = new-object System.Drawing.Size(75,23)
$olButton.Text = "Select file"
$olButton.Add_Click({openLog})
$form.Controls.Add($olButton)
# Add FileName Lable
$fnFileNamelableBox = new-object System.Windows.Forms.Label
$fnFileNamelableBox.Location = new-object System.Drawing.Size(110,25)
$fnFileNamelableBox.forecolor = "MenuHighlight"
$fnFileNamelableBox.size = new-object System.Drawing.Size(200,20)
$form.Controls.Add($fnFileNamelableBox)
# Add Refresh Log file Button
$refreshButton = new-object System.Windows.Forms.Button
$refreshButton.Location = new-object System.Drawing.Size(390,29)
$refreshButton.Size = new-object System.Drawing.Size(75,23)
$refreshButton.Text = "Refresh"
$refreshButton.Add_Click({Populatetable})
$form.Controls.Add($refreshButton)
# Show Summary CheckBox
$SsumBox = new-object System.Windows.Forms.CheckBox
$SsumBox.Location = new-object System.Drawing.Size(510,7)
$SsumBox.Size = new-object System.Drawing.Size(200,20)
$SsumBox.Checked = $true
$SsumBox.Add_Click({if ($SsumBox.Checked -eq $true){$dgDataGrid.DataSource = $sumTable}
else {$dgDataGrid.DataSource = $logTable}})
$SsumBox.Text = "Show Summary"
$form.Controls.Add($SsumBox)
$dgDataGrid = new-object System.windows.forms.DataGrid
$dgDataGrid.AllowSorting = $True
$dgDataGrid.Location = new-object System.Drawing.Size(12,81)
$dgDataGrid.size = new-object System.Drawing.Size(1024,750)
$form.Controls.Add($dgDataGrid)
$form.topmost = $true
$form.Add_Shown({$form.Activate()})
$form.ShowDialog()