Less Thrashing; More Sorting Queries (Part IIl)

The cruft below demonstrates (somewhat) how to effectively interrogate  70k events from Windows  with PS 3.0. It presumes you are using 'auditpol' to your advantage. When querying that many events, I keep a check on memory in the title bar with this function:


function Global:set-titleMemoryStats {

# With 3.0 Runspace
$set_title=

{

function Global:Set-title {
$PSID=([System.Diagnostics.Process]::GetCurrentProcess()).Id
$MemStats=ps -id $PSID | Select `
@{Name='ThreadCount';Expression={($_.Threads).count}}, `
@{Name='WorkSetMB';Expression={[int](($_.WorkingSet64)/1MB)}}, `
@{Name='VirMemMB';Expression={[int](($_.VirtualMemorySize64)/1MB)}}, `
@{Name='PriMemMB';Expression={[int](($_.PrivateMemorySize64)/1MB)}}, `
@{Name='PagedMemMB';Expression={[int](($_.PagedMemorySize64)/1MB)}}, `
@{Name='NonPagedMemKB';Expression={[int](($_.NonPagedSystemMemorySize64)/1KB)}}

$Title=write "Last_Title_Stats: Time: $([datetime]::now) Version: $((get-host).Version.Major) SessionHours: $([int]([datetime]::now - (ps -id $psid).Starttime).totalhours) Memory: $($Memstats) GC_MB: $([int]([GC]::gettotalmemory(1)/1MB))"
[console]::set_title($Title)
}
while(1) {set-title;sleep -s 5}
}

$ST_Runspace = [PowerShell]::Create().AddScript($set_title)
$Begin_Set_Title = $ST_Runspace.BeginInvoke()

# To stop all of this...
# $ST_Runspace.runspace
# $Stop_Set_Title = $ST_Runspace.Stop()
# $Dispose_Set_Title = $ST_Runspace.Dispose()
}

On with the demonstration.  The filterhashtable parameter will allow function with one array per query. (e.g. 'ID=@(5156..5158)'). However, we could select out specific IDs for each log included. I will leave it to the reader to decide which security audit subcategories ('auditpol /get /category:*') are imortant. I am focusing on 5156, 5157, 5158 (e.g. 'Filtering Platform Connection', 'Filtering Platform Policy Change'). Here is the complete list of security events from Microsoft. 

After some experimentation, I use the following PSObject based query to increase the query speed. Honestly, I am not sure why it works so well.:

function Global:Search-EventLog
{
    [CmdletBinding(SupportsPaging = $true)]
    param(
$maxSecurity=5000,
$maxSystem=1000,
$maxApplication=1000
 )

$Elements=New-Object PSObject -Property @{
Security_515X=get-winevent -ea 0 -max $maxSecurity -filterhashtable @{logname='Security';ID=@(5156..5158)}
System=get-winevent -ea 0 -max $maxSystem -filterhashtable @{logname='System'}
Application=get-winevent -ea 0 -max $maxApplication -filterhashtable @{logname='Application'}
}

if ($All_Events) {rv -ea 0 All_Events}
$global:Events= foreach ($element in $Elements) {$element}
[array]$HAElements="Security_515X","System","Application"
foreach ($element in $HAElements){$All_Events+=$Events.$element}
$Global:EventLog=$All_Events | Sort -desc -property TimeCreated
}


#70K events in 5 min 14 seconds!
measure-command {Search-EventLog 50000 10000 10000}


Days              : 0
Hours             : 0
Minutes           : 5
Seconds           : 14
Milliseconds      : 790
Ticks             : 3147907742
TotalDays         : 0.00364341173842593
TotalHours        : 0.0874418817222222
TotalMinutes      : 5.24651290333333
TotalSeconds      : 314.7907742
TotalMilliseconds : 314790.7742

# Now we find the IDs and sort them by count.
$EventLog | group -property ID -noelement | Sort -desc -property Count

Count Name
----- ----
24669 5156
23893 5158
 7167 7036
 2331 257
 2070 0
 1651 1035
 1438 5157
  743 1
...

# Now we sort them by ProviderName.
$EventLog | group -property ProviderName -noelement | Sort -desc -property Count | ft -auto -wrap

Count Name
----- ----
50000 Microsoft-Windows-Security-Auditing
 7972 Service Control Manager
 2501 MsiInstaller
 2352 SampleCollector
 1182 gupdate
  345 Windows Error Reporting
  342 Microsoft-Windows-Security-SPP
  317 Microsoft-Windows-CAPI2
  315 Microsoft-Windows-Kernel-General
  309 gusvc
  289 Microsoft-Windows-Kernel-Power
  278 Microsoft-Windows-Power-Troubleshooter
  257 VSS
  250 Microsoft-Windows-RestartManager
  248 ESENT
  235 Microsoft Antimalware
  173 Microsoft-Windows-WMI
  153 Microsoft-Windows-WindowsUpdateClient
  134 System Restore
  129 Microsoft-Windows-DNS-Client
  125 Microsoft-Windows-Winlogon
 ...
# Top Events with a count of more than 1000

$EventLog | group -property LogName,ID,ProviderName -noelement | Sort -desc -property Count | where Count -gt 1000 | ft -auto | more

Count Name
----- ----
31874 Security, 5156, Microsoft-Windows-Security-Auditing
17923 Security, 5158, Microsoft-Windows-Security-Auditing
 7154 System, 7036, Service Control Manager
 2239 Application, 10010, Microsoft-Windows-RestartManager
 1236 Application, 10001, Microsoft-Windows-RestartManager
 1233 Application, 10000, Microsoft-Windows-RestartManager
 1099 Application, 10005, Microsoft-Windows-RestartManager


# Sorted Count of Events Per Provider
$EventLog | ? {$_.ProviderName -eq "Microsoft-Windows-Security-Auditing"} | group -property ID -noelement | Sort -desc -property Count | ft -auto -wrap

Count Name
----- ----
24669 5156
23893 5158
 1438 5157

# We want to look at the Event IDs for one specific provider.
$EventLog | ? {$_.ProviderName -eq "Service Control Manager"} | group -property ID -noelement | Sort -desc -property Count | ft -auto -wrap

Count Name
----- ----
 7161 7036
  398 7040
  286 7042
   95 7011
   18 7045
    5 7039
    3 7030
    3 7009
    2 7000
    1 7034


# Okay! Just the unique messages from all events for the "Service Control Manager" provider:
[array[]]$a=$EventLog | ? {$_.ProviderName -eq "Service Control Manager"} | group -property ID -noelement | Sort -desc -property Count
 $a.Name | Sort -desc
7045
7042
7040
7039
7036
7034
7030
7011
7009
7000
[array[]]$b=$a.Name | Sort -desc
$c=foreach ($i in $b) {$EventLog | ? {$_.ID -eq "$i"} | Select -unique | group -property Message -noelement}
$c | ft -auto -wrap

Count Name
----- ----
    1 A service was installed in the system.

      Service Name:  Google Software Updater
      Service File Name:  "C:\Program Files (x86)\Google\Common\Google Updater\GoogleUpdaterService.exe"
      Service Type:  user mode service
      Service Start Type:  disabled
      Service Account:  LocalSystem
    1 The TCP/IP NetBIOS Helper service was successfully sent a stop control.

       The reason specified was: 0x40030011 [Operating System: Network Connectivity (Planned)]

       Comment: None
    1 The start type of the Windows Modules Installer service was changed from auto start to demand start.
    1 A service process other than the one launched by the Service Control Manager connected when starting the Google Update Service (gupdate) service.  The Service
      Control Manager launched process 8096 and process 8324 connected instead.

        Note that if this service is configured to start under a debugger, this behavior is expected.
    1 The Google Update Service (gupdate) service entered the stopped state.
    1 The Eventlog to Syslog service terminated unexpectedly.  It has done this 1 time(s).
    1 The PST Service service is marked as an interactive service.  However, the system is configured to not allow interactive services.  This service may not function
      properly.
    1 A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VcmIAlzMgr service.
    1 A timeout was reached (30000 milliseconds) while waiting for the Windows Error Reporting Service service to connect.
    1 The Windows Search service failed to start due to the following error:
      The service did not respond to the start or control request in a timely fashion.



# Some methods to display query event IDs from regular expressions (see 'man about_regular_expressions')


$EL_4xxx=$EventLog | ? {$_.ID -match '^4...'}
$EL_4xxx | ft -auto -wrap TimeCreated,ID,RecordID,ProviderName, `
@{Name="Message[100]"; Expression={($_.Message).substring(0,100)}}


TimeCreated             Id RecordId ProviderName                      Message[100]
-----------             -- -------- ------------                      ------------
7/12/2012 11:41:31 AM 4000    87634 Microsoft-Windows-WLAN-AutoConfig
...

[array[]]$a=$EL_4xxx | Select TimeCreated,ID,RecordID,ProviderName, `
@{Name="Message[100]"; Expression={($_.Message).substring(0,100)}}
$a.'Message[100]'| sls 'certificate'

Successful auto update of disallowed certificate list with effective date: Thursday, June 21, 2012 4
Successful auto update of third-party root certificate:: Subject:
Successful auto property update of third-party root certificate:: Subject:
Successful auto update of third-party root certificate:: Subject:
...

$EL_1xxx=$EventLog | ? {$_.ID -match '^1...'}
$EL_1xxx | ft -auto -wrap TimeCreated,ID,RecordID,ProviderName, `
@{Name="Message[100]"; Expression={($_.Message).substring(0,100)}}


TimeCreated            Id RecordId ProviderName                                Message[100]
-----------            -- -------- ------------                                ------------
9/5/2012 3:32:06 PM 10000    92858 Microsoft-Windows-DriverFrameworks-UserMode A driver package which uses user-mode driver framework version 1.9.0 is being installed
                                                                               on device US

...

$EL_8xxx=$EventLog | ? {$_.ID -match '^8...'}
$EL_8xxx | ft -auto -wrap TimeCreated,ID,RecordID,ProviderName, `
@{Name="Message[100]"; Expression={($_.Message).substring(0,100)}}

TimeCreated            Id RecordId ProviderName   Message[100]
-----------            -- -------- ------------   ------------
8/25/2012 9:08:21 PM 8194    32462 System Restore Successfully created restore point (Process = C:\Windows\system32\svchost.exe -k netsvcs; Descriptio
...