Avoiding XPath: Part VI

Updated some 1/11/2020 -RMF

So this piece will begin a discussion about grokking Windows Security Event Logs inside rdata.table and Postgres. Use auditpol to set up kernel logging. We recall that to convert your EVTX archived security logs to CSV we need a Powershell function as below:

# Powershell Memory and CPU intensive
Function Convert-Logs3 {
[cmdletbinding()]
Param(
$filelist=$NULL
)
$filelist | foreach-object {
#Note that I am only getting four columns of data
Get-WinEvent -Path "$PSItem"| Select RecordID,ID,TimeCreated, Message | export-csv -notypeinformation -path $(write "$PSItem.csv");
[System.gc]::collect();
}
}

Convert-Logs3 "Archive-Security-2019-12-23-04-25-01-062.evtx"

#R
library(data.table)
d <- fread("Archive-Security-2019-12-23-04-25-01-062.evtx.csv")
h <- gsub('\n\t',' ',d$Message,fixed=TRUE)
h <- gsub('\n\n',' ',h,fixed=TRUE)
h <- gsub('\t\t',' ',h,fixed=TRUE)
h <- gsub('\n',' ',h,fixed=TRUE)
h <- gsub('\t',' ',h,fixed=TRUE)
h <- gsub('%%','',h,fixed=TRUE)
h <- gsub('\r','',h,fixed=TRUE)
h <- gsub('\r \r','',h,fixed=TRUE)
d$Message <- h

This produces row headings with data types:

names(d)
[1] "RecordId"    "Id"          "TimeCreated" "Message"
 
d[,sapply(.SD,class)]
   RecordId          Id TimeCreated     Message
  "integer"   "integer" "character" "character" 


# where you can use sample rdata.table queries like this:

d[,.N,.(Id,Message=substr(Message,0,100))][order(-N)]
d[Id == 4688,.(Id,TimeCreated,Message=substr(Message,275,375))]
d[Id == 4672 & !duplicated(substr(TimeCreated,0,11)),
.(Id,TimeCreated,Date=substr(TimeCreated,0,11),Message=substr(Message,400,500))]
d[Id == 4624 & grepl("S-1-0-0",Message),.(Id,Message=substr(Message,200,400))]
d[Id ==  4907,.(Id,TimeCreated,paste0(substr(Message,0,50),substr(Message,250,375)))]

There is probably much more you can do with Security.evtx files with mlr3 or rdata.table

# and then...I write out the file to be imported into Postgres:
write-csv(d, "Archive-Security-2019-12-23-04-25-01-062.csv")

Avoiding XPath: Part IV

 Full source  of my unpolished cruft is far below.  If you are going to pull fields out of the Message in Windows Event log without Xpath or XML, how are you going to do it in Powershell 4.0? I will remind you of what the Message field looks like:


Message              : The Windows Filtering Platform has permitted a connection.

                       Application Information:
                           Process ID:        3116
                           Application Name:    \device\harddiskvolume3\users\rferrisx\appdata\local\chromium\application\chrome.exe

                       Network Information:
                           Direction:        Outbound
                           Source Address:        192.168.0.11
                           Source Port:        2094
                           Destination Address:    8.247.65.200
                           Destination Port:        80
                           Protocol:        6

                       Filter Information:
                           Filter Run-Time ID:    211332
                           Layer Name:        Connect
                           Layer Run-Time ID:    48

So I can get to these fields with 'properties':

Less Thrashing; Better Queries (Part V)

# Updated: 7:01 AM 1/19/2014  -RMF

# Using [System.Diagnostics.EventLog] for Powershell 3.0 Beta
# Code
# Clearing variable types 
rv -ea 0 i;
rv -ea 0 var
$var=@("a","b","c","d","e"); foreach ($i in $var) {rv -ea 0 $i}
 #Creating $a specific to the 'GetEventLogs()' 
 # method for [System.Diagnostics.EventLog]
 $a=[System.Diagnostics.EventLog]::GetEventLogs()
 $a | gm -s
 # List the event logs
 $a

 # Creating $a as generic to the .NET class; Querying active
 # Eventlog for a local(or remote?)computer name:
 $a=[System.Diagnostics.EventLog]
 $a::GetEventLogs("rmfvpc")
 $a::GetEventLogs("rmfvpc") | gm -s

 # Creating $B as the result of mahine specific
 #'GetEventLogs()' query
 $b=$a::GetEventLogs("rmfvpc")
 $b | gm -s
 $b | gm -f

 # Using $B to get a specific method for a specific log (e.g. 
 # Array[10]) for specific configuration method (e.g. 
 # 'get_OverflowAction()')
 $b[10]
 $b[10].get_OverflowAction()

 # Choose the Security Log
 $C = $B | Where Log -eq Security

 # This retrieves all Entries before returning the first index.
 $c[0].get_Entries()[0]
 $c[0].get_Entries()[0] | gm -f

 #Returns select entries and then select EventIDs for such.
 $c[0].get_Entries()[100..110]
 $c[0].get_Entries()[100..110]
 $c[0].get_Entries()[100..110].get_EventID()

 # Number of Events Logs; Number of total events for a 
 # specific Event Log.
 $c[0].count
 $c[0].Entries.count

 # Returns First and Last Events
 $lc = $($c[0].Entries.count - 1) 
 $c[0].get_Entries()[0,$($c[0].Entries.count - $lc)]

 # Creates a DateTime variable;Returns number of days 
 # between first and last events
 ($c[0].get_Entries()[0,$lc]).TimeGenerated
 $TG=($c[0].get_Entries()[0,$lc]).TimeGenerated
 $TG  | gm -s
 $TG[1]-$TG[0]
 ($TG[1]-$TG[0]).Days

 # Returns select sorted information last 1000 entries
 $d=($c[0].get_Entries())[0..999]| Select EventID,Message
 $d.count
 $d[0..9] | ft -auto -wrap
 $d | group -property EventID -noelement | sort -desc -property Count
 $e= ($d | group -property Message -noelement | sort -desc -property Count)
 $e.count
 $e[0..9] | ft -auto -wrap

Less Thrashing; More Sorting Queries (Part IIl)

The cruft below demonstrates (somewhat) how to effectively interrogate  70k events from Windows  with PS 3.0. It presumes you are using 'auditpol' to your advantage. When querying that many events, I keep a check on memory in the title bar with this function:


function Global:set-titleMemoryStats {

# With 3.0 Runspace
$set_title=

{

function Global:Set-title {
$PSID=([System.Diagnostics.Process]::GetCurrentProcess()).Id
$MemStats=ps -id $PSID | Select `
@{Name='ThreadCount';Expression={($_.Threads).count}}, `
@{Name='WorkSetMB';Expression={[int](($_.WorkingSet64)/1MB)}}, `
@{Name='VirMemMB';Expression={[int](($_.VirtualMemorySize64)/1MB)}}, `
@{Name='PriMemMB';Expression={[int](($_.PrivateMemorySize64)/1MB)}}, `
@{Name='PagedMemMB';Expression={[int](($_.PagedMemorySize64)/1MB)}}, `
@{Name='NonPagedMemKB';Expression={[int](($_.NonPagedSystemMemorySize64)/1KB)}}

$Title=write "Last_Title_Stats: Time: $([datetime]::now) Version: $((get-host).Version.Major) SessionHours: $([int]([datetime]::now - (ps -id $psid).Starttime).totalhours) Memory: $($Memstats) GC_MB: $([int]([GC]::gettotalmemory(1)/1MB))"
[console]::set_title($Title)
}
while(1) {set-title;sleep -s 5}
}

$ST_Runspace = [PowerShell]::Create().AddScript($set_title)
$Begin_Set_Title = $ST_Runspace.BeginInvoke()

# To stop all of this...
# $ST_Runspace.runspace
# $Stop_Set_Title = $ST_Runspace.Stop()
# $Dispose_Set_Title = $ST_Runspace.Dispose()
}

Get-Winevent Part IV: Querying the Event Log for 'Filtering Platform Connection' Information (Part A)

The command:

'auditpol /set /subcategory:"Filtering Platform Connection" /success:enable /failure:enable'

enables the "Filtering Platform Connection" security counter on Windows 7. The "Filtering Platform Connection" gives your event logs access to the following counters:

Filtering Platform Connection           Success and Failure

• Object Access Filtering Platform Connection 5150 The Windows Filtering Platform has blocked a packet. Windows 7, Windows Server 2008 R2
• Object Access Filtering Platform Connection 5151 A more restrictive Windows Filtering Platform filter has blocked a packet. Windows 7, Windows Server 2008 R2
• Object Access Filtering Platform Packet Drop 5152 The Windows Filtering Platform blocked a packet. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Packet Drop 5153 A more restrictive Windows Filtering Platform filter has blocked a packet. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Connection 5154 The Windows Filtering Platform has permitted an application or service to listen on a port for incoming connections. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Connection 5155 The Windows Filtering Platform has blocked an application or service from listening on a port for incoming connections. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Connection 5156 The Windows Filtering Platform has allowed a connection. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Connection 5157 The Windows Filtering Platform has blocked a connection. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Connection 5158 The Windows Filtering Platform has permitted a bind to a local port. Windows Vista, Windows Server 2008
• Object Access Filtering Platform Connection 5159 The Windows Filtering Platform has blocked a bind to a local port. Windows Vista, Windows Server 2008

This script, which uses some Powershell 3.0 features, produces the output far below (abbreviated) by parsing the output from EventID 5156 ("allowed connection"). The loops are structured to allow 'findstr' to dig out 'subfield' information. 'Select -unique' functions to find unique addresses (or ports):

[array]$a=Get-WinEvent -FilterHashTable @{LogName='Security';ID=5156;StartTime=$StartTime}
$UDA_count=$a.count
[array[]]$b=$a.Message | findstr 'Destination' | findstr 'Address'
$Global:UDestAddress=($b | Select -unique) | sort

The script takes an extremely long time to run on my five core laptop. These scripts (1,2) are optimized a bit more to search for only 5156 Events. The global variables in the script would be suitable for parsing against lists of allowed ports, allowed or blocked IPs. The Script can be used as a format for other counters as well. Several features from Powershell 3.0 are used in this script including the ability of Powershell 3.0 to 'automatically unroll' an entire array for a certain property (e.g. '[array[]]$b=$a.Message'). I could dearly use a much faster Powershell method to dig 'subfield' information out of the Message field than double piping that information to 'findstr'. The issue is that a single day of network activity generates ten of thousands of kernel security counters.  An alternative to limit the amount of information returned might be to use the '-max' [number of events] parameter:

Muxing AccessControl and FileInfo objects

Most of us know the members (partially printed at bottom) of System.Security.AccessControl and System.IO.FileInfo. And most of us know they both share the PS* NoteProperty items:

• PSChildName                     NoteProperty   System.String PSChildName=test.txt
• PSDrive                         NoteProperty   System.Management.Automation.PSDriveInfo PSDrive=C
• PSParentPath                    NoteProperty   System.String PSParentPath=Microsoft.PowerShell.Core\FileSystem::C:\
• PSPath                          NoteProperty   System.String PSPath=Microsoft.PowerShell.Core\FileSystem::C:\test.txt
• PSProvider                      NoteProperty  

Get-WinEvent, EventLogs, ETL, Providers on Win7 Part III

Microsoft has exposed substantial providers since XP. With Windows 7, Microsoft has increased the number of providers substantially over previous versions of Windows and added 'netsh trace' functionality to enable tracing, conversion, batching of these kernel level counters. In the commands below, I have mixed cmd shell, powershell, cygwin cmds to parse ETL files. In  general, parsing etl files with 'get-winevent' and powershell takes a while...  You can understand 'netsh' filtering best with 'netsh trace show CaptureFilterHelp', however I recommend setting your 'netsh trace start maxSize=' parameter at 150 MB or less. (The default is an almost unworkable 250MB.)

Get-Winevent Part III: Querying the Event Log for Logons (Part D)

In Part A of this series ('Get-Winevent Part III Querying the Event Log for logons'), I worked with the 'where-object' cmdlet to filter through properties of specific logon event types. In Part B, I used '-filterhashtable' and 'findstr' to more quickly dig into the message field of logon events, utlimately producing a spreadsheet or database format of those events. In Part C, I presented code that enumerates all provider types for these events.  Then I used '-filterhashtable' with an array of multiple security EventIDs whose select 'Message' fields I searched with 'findstr' for specific properties relating to logons.  In this post (Part D),  I pull this all together, creating a timeline of multiple security EventIDs whose select 'Message' fields I pump into a spreadsheet for further analysis.

Get-Winevent Part III: Querying the Event Log for Logons (Part A)

The following is a digression on using Powershell's where-object (filter) to query System and Administrative events with 'Get-WinEvent'.  I like this method of querying the event logs because it is "pipeline" oriented and allows me to re-use/amend/copy previous syntax.  I was having some concern understanding a mysterious problem: my Windows 7 PC spontaneously un-sleeps itself and seemingly commences a log-on. I wanted to understand why this happened and if there was evidence of ex-filtration or malware.
  

Get-WinEvent, EventLogs, ETL, Providers on Win7

'Get-WinEvent' in Powerhsell 2 when combined with ETL on Windows 7 allows exceptional event log queries. This function allows the administrator to create an array of all Event Logs and sort by 'time created' all those records created in the last (1) day:



function global:LatestLogEntries
{
   [CmdletBinding()]
   Param(
       [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
       [int32] $param1,
       [string] $ErrorActionPreference="silentlycontinue"
            )


$LogNames=(Get-Winevent -listlog  * )
$goback = (get-date) - (new-timespan -days $param1 )
$LogNames | % {get-winevent -FilterHashTable  @{LogName=$_.LogName;StartTime=$goback}}
}

time stamping windows directory and file names

This is something I have blogged about before, but I thought it worth posting again.  Special characters need to be eliminated to create a time stamp that can be used as a Windows file name. The `date` program in Unix has a number of very useful options for this.  Windows cmd shell is more limited. This is what I use:

:: rtime.cmd
@echo off

set realdate=%date:/=.%
set realdate=%realdate:* =%
set realtime=%time::=.%
set realtime=%realtime:* =%
set timestamp=%realdate%.%realtime%
echo %timestamp%

This command script uses 'variable substitution' from the set command to remove special characters (e.g. :  / ) unacceptable as Windows file or directory names . This line:
set timestamp=%realdate%.%realtime%


can be changed as needed for more CSV compatible logging:
set timestamp="%realdate%","%realtime%"


Once cached, it runs pretty fast and is suitable for lightweight logging:

$ time /cygdrive/C/Security/rtime.cmd
06.02.2010.11.04.05.99

real    0m0.202s
user    0m0.015s
sys     0m0.031s

$ time /cygdrive/C/Security/rtime.cmd
06.02.2010.11.04.12.65

real    0m0.062s
user    0m0.000s
sys     0m0.015s

$ time /cygdrive/C/Security/rtime.cmd
06.02.2010.11.04.14.68

real    0m0.062s
user    0m0.000s
sys     0m0.015s

piping tcpdump output to lsof

This simple Bash script will output the lsof end of any foreign network connection:
[Set to the interface of your choice]
while [ 1 ]
        do
                for i in `tcpdump -i rl0 -c 1 -l dst $(hostname) | awk '{print $2}' | awk -F"." '{print $1"."$2"."$3"."$4}'`
                         do lsof -i@$i
                done
done
with time/date stamp added and headers removed:
while [ 1 ]

        do
                for i in `tcpdump -i rl0 -c 1 -l dst $(hostname) | awk '{print $2}' | awk -F"." '{print $1"."$2"."$3"."$4}'`
                         do echo `date -u` `lsof -i@$i | grep -v PID`
                done
done

Run like this:
./tcp_lsof.sh >> tcp.lsof.log &

the script produces output like this:

COMMAND   PID     USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
sshd    18392 rferrisx    5u  IPv4 0xd699ac80      0t0  TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:13974 (ESTABLISHED)
sshd    29850     root    5u  IPv4 0xd699ac80      0t0  TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:13974 (ESTABLISHED)
or
Wed May 26 15:22:06 UTC 2010 sshd 9448 root 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:15729 (ESTABLISHED) 
sshd 29734 rferrisx 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:15729 (ESTABLISHED)
Wed May 26 15:22:07 UTC 2010 sshd 9448 root 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:15729 (ESTABLISHED) 
sshd 29734 rferrisx 5u IPv4 0xd699ac80

A prototype test harness...but needs lots of work

I have spent too much time here in the last few days working on a test harness for live network files in Vista. As a prototype, what I have written may be useful. However, numerous problems were uncovered.  The idea was this: At any moment they are a discoverable set of files that are being accessed by the network. In theory, you should be able to list those files and then query them for their integrity. The heart of this is something like:  
  
icacls %dir_file%                                                                         &( 
if /I [%filetype% EQU [regular sfc /verifyfile=%dir_file% ) &(
if /I [%filetype% EQU [regular accesschk -qv %dir_file% ) &(
if /I [%filetype% EQU [regular sigcheck -q %dir_file% )

Definitely some useful information is returned. But the project will have to be rewritten in a faster language with better string support. Interesting to see what information it did return. Like the file - C:\Windows\System32\nsi.dll - below.

 Running icacls, sfc, accesschk, sigcheck for FileType,FileID,Path: regular 1220: "C:\Windows\System32\nsi.dll "
filetype=regular
C:\Windows\System32\nsi.dll NT SERVICE\TrustedInstaller:(F)
BUILTIN\Administrators:(RX)
NT AUTHORITY\SYSTEM:(RX)
BUILTIN\Users:(RX)

Successfully processed 1 files; Failed processing 0 files

Windows Resource Protection could not perform the requested operation.
C:\Windows\System32\nsi.dll
Medium Mandatory Level (Default) [No-Write-Up]
RW NT SERVICE\TrustedInstaller
FILE_ALL_ACCESS
R BUILTIN\Administrators
FILE_EXECUTE
FILE_LIST_DIRECTORY
FILE_READ_ATTRIBUTES
FILE_READ_DATA
FILE_READ_EA
FILE_TRAVERSE
SYNCHRONIZE
READ_CONTROL
R NT AUTHORITY\SYSTEM
FILE_EXECUTE
FILE_LIST_DIRECTORY
FILE_READ_ATTRIBUTES
FILE_READ_DATA
FILE_READ_EA
FILE_TRAVERSE
SYNCHRONIZE
READ_CONTROL
R BUILTIN\Users
FILE_EXECUTE
FILE_LIST_DIRECTORY
FILE_READ_ATTRIBUTES
FILE_READ_DATA
FILE_READ_EA
FILE_TRAVERSE
SYNCHRONIZE
READ_CONTROL
c:\windows\system32\nsi.dll:
Verified: Signed
Signing date: 8:08 AM 1/19/2008
Strong Name: Unsigned
Publisher: Microsoft Corporation
Description: NSI User-mode interface DLL
Product: Microsoft« Windows« Operating System
Version: 6.0.6001.18000
File version: 6.0.6001.18000 (longhorn_rtm.080118-1840)

tcpslice II

More uses for tcpslice, ipsumdump, BASH 4.1 :

[This gives you today's top source IP and source IP Port combination: 


/usr/sbin/tcpslice `date +%Y"y"%m"m"%d"d"` $BASH_ARGV | ipsumdump --no-headers -sD - 

./todays_dump.sh MarApr.snort.in.tcpd | sort -nr | uniq -c | sort -nr
     13 85.144.201.237 7959
      3 95.179.99.147 5900
      3 64.206.157.2 23
      3 222.45.112.59 8085
      3 109.187.8.70 5900
      2 98.247.214.152 23 ...

This gives you today's top source IP and source IP location:


/usr/sbin/tcpslice `date +%Y"y"%m"m"%d"d"` $BASH_ARGV |
for i in `ipsumdump --no-headers -s -`
     do echo $i : $(printf "%s" `./geoip.sh $i | awk -F":" '{print $2}' | awk -F"," '{print $1","$2","$3}' ` ) 
done 

./tgeodump.sh MarApr.snort.in.tcpd | sort -nr | uniq -c | sort -nr
     13 85.144.201.237 : NL,07,Amsterdam
     12 222.45.112.59 : CN,22,Beijing
      4 222.215.230.49 : CN,32,Chengdu
      3 95.179.99.147 : RU,43,Lipetsk
      3 64.206.157.2 : US,NH,Nashua
      3 109.187.8.70 : IPAddressnotfound,,
      2 98.247.214.152 : US,WA,Bothell ...

where 'geoip.sh' is:
geoiplookup -f /usr/local/share/GeoIP/GeoLiteCity.dat $1


I note that file names like this '08Mar1142PST2010.in.1268074842' don't process through tcpslice.

tcpslice

Tcpslice is a useful tool from LBL network group that allows you to carve up a large pcap file format into time slices. 

To look at the start and finish time stamps of the entire pcap file in various time formats:

tcpslice -r Marchrferrisx.snort.in 

Marchrferrisx.snort.in  Mon Mar  8 11:08:09 2010        Mon Apr  5 09:09:37 2010

tcpslice -t Marchrferrisx.snort.in

Marchrferrisx.snort.in  2010y03m08d11h08m09s660222u     2010y04m05d09h09m37s390876u

tcpslice -R Marchrferrisx.snort.in

Marchrferrisx.snort.in  1268075289.660222       1270483777.390876

To return data from a particular time slice to a file with BPF filters use syntax like this: 

tcpslice 1257347146.060 1257347146.061 inputFile.tcpd | tcpdump -r - -w outputFile.tcpd 'host 192.168.1.175'

(Check out bothunter logs for more examples like this..)

In this example, I want all the packets that are not IPv6 for one date:

/usr/sbin/tcpslice 2010y04m05d Marchrferrisx.snort.in | /usr/sbin/tcpdump -r - 'not(ip6)' | less

reading from file -, link-type EN10MB (Ethernet)
01:06:17.290514 IP 125.141.195.190.35460 > 192.168.0.12.ssh: S 1607742099:1607742099(0) win 65535 
01:40:16.181816 IP c-98-247-214-152.hsd1.wa.comcast.net.catchpole > 192.168.0.12.telnet: SWE 498716114:498716114(0) win 5840
01:40:19.172942 IP c-98-247-214-152.hsd1.wa.comcast.net.catchpole > 192.168.0.12.telnet: SWE 498716114:498716114(0) win 5840
01:44:01.423708 IP hn.kd.ny.adsl.x11 > 192.168.0.12.ms-sql-s: S 833421312:833421312(0) win 16384
03:37:06.073237 IP 75.125.252.76.http > 192.168.0.12.48532: S 1175613974:1175613974(0) ack 143375003 win 14420
04:07:03.019711 IP 222.45.112.59.12200 > 192.168.0.12.ssm-els: S 363594672:363594672(0) win 8192 ...

Now I want all ms-sql-s destination packets from the ingress pcap that are not IPv6 for all of March:

/usr/sbin/tcpslice 2010y04m01d 2010y04m31d Marchrferrisx.snort.in | /usr/sbin/tcpdump -r - -n 'dst port(1433)'
reading from file -, link-type EN10MB (Ethernet)
18:33:42.614843 IP 125.46.78.100.x11 > 192.168.0.12.ms-sql-s: S 908984320:908984320(0) win 16384
23:38:50.771853 IP 61.183.172.35.x11 > 192.168.0.12.ms-sql-s: S 47316992:47316992(0) win 16384
03:35:18.351118 IP 121.12.125.7.x11 > 192.168.0.12.ms-sql-s: S 640548864:640548864(0) win 16384
11:09:45.631103 IP 218.61.127.71.x11 > 192.168.0.12.ms-sql-s: S 1613627392:1613627392(0) win 16384
00:47:21.207593 IP 218.90.163.66.x11 > 192.168.0.12.ms-sql-s: S 648937472:648937472(0) win 16384
08:56:05.732622 IP 61.183.172.35.x11 > 192.168.0.12.ms-sql-s: S 47316992:47316992(0) win 16384
18:06:53.798198 IP 59.51.114.39.x11 > 192.168.0.12.ms-sql-s: S 648937472:648937472(0) win 16384 ...
 

Something similar, but a little cleaner, can be done with ipsumdump:


/usr/sbin/tcpslice 2010y04m01d 2010y04m31d Marchrferrisx.snort.in | ipsumdump -tsD | grep -w 1433
 
1270172022.614843 125.46.78.100 1433 
1270190330.771853 61.183.172.35 1433 
1270204518.351118 121.12.125.7 1433 
1270231785.631103 218.61.127.71 1433 
1270280841.207593 218.90.163.66 1433 
1270310165.732622 61.183.172.35 1433 
1270343213.798198 59.51.114.39 1433 ...

Advanced Persistent Threat Part III

It certainly is possible to examine host or network outbound conversations.    But we then have to determine which outbound conversations are legitimate.   Current AV software attempts to block access to potentially 'known dangerous' or 'pre-determined dangerous'  malware sites but such judgements are apparently failing to prevent APT from sending stolen data to weigh stations.  On OpenBSD if we are looking at  outbound connections, we might sniff as thus using Snort:

/usr/local/bin/snort -D -vdeXX -l . -L `date "+%d%b%H%S%Z%Y.out"` -i dc0  'port not(whois or domain or router) and not(broadcast or arp) and not(dst net 192.168.0.0/24 or 224.0.0.0/24 or 239.0.0.0/8)' 

On Vista, we might have two interfaces (wired and wireless) we need to examine:

start /min cmd /c C:\snort\bin\snort.exe  -vdeXX -l .  -i 1  port not(whois or domain or router or 5353) and not(broadcast or arp) and not(dst net 192.168.0.0/24 or 224.0.0.0/24 or 239.0.0.0/8)
start /min cmd /c C:\snort\bin\snort.exe  -vdeXX -l .  -i 2  port not(whois or domain or router or 5353) and not(broadcast or arp) and not(dst net 192.168.0.0/24 or 224.0.0.0/24 or 239.0.0.0/8)

We can look at the logs. And we are surprised by the number of outbound connections we make:
C:\Snort\bin>snort -v -q -r snort.log.1266372570 | find "->" | gawk -F"->" '{print $2}' | sort /R | uniq -c | sort /R
    327  74.125.103.208:80
    133  74.202.67.83:80
    105  216.35.221.76:80
    100  198.104.200.154:80
     51  72.21.91.19:80
     32  96.17.70.50:80
....

Perhaps one solution to APT would be some real time co-ordination between sites suspected of being data theft transfer stations and real-time (firewall or host) blocking of the data-transfer to those hosts/servers.  This type of solution has some headwind but may need to be implemented on a individual or corporate basis to prevent "incidental blacklisting".  Other solutions might include:

(1) real time packet examination of data for critical or sensitive information
(2) heuristic detection of data flows that seems 'abnormal'
(3) heuristic detection of file access that seems 'abnormal'

The industry awaits such solutions.

Advanced Persistent Threat

The news on  "Advanced Persistent Threat" has been broken in a big way by Google and the recent Mandiant report.  More comments will follow at a later date.  But some occur to me now:

(1) Our current desktop and server Operating Systems are not secure.
(2) Computer networks are insecure for most organizations and at many levels.
(3) Digital data can no longer be protected against a determined foe.
(4) Security researchers and visionaries should receive more funding. Lots.

Order and read the Mandiant Report. Then imagine what a resourced foe could do if they believed the security of their nation-state depended upon seemless corporate intrusions.  Now imagine those techniques automated and in the wild.  In order for the world to have safe computing systems, our government and industry needs to sponsor more research and decriminalize vulnerability research. Otherwise, no data will ever be secret or protected again.

Securing Digital Content: Part I

I will post no code for this blog entry so that I can answer the question of a friend of mine :  How does a normal person take reasonable steps to safeguard sensitive digital content in this day of repeated sophisticated instrusions, penetrations, institutionalized hacking and institutionalized snooping? This is a long subject that would require more than just one blog entry. Here are some (random or not) thoughts: 

Part I Strategies
 I would answer the strategy for maintaining content security like this:
(1) Assume data loss or data theft. Develop a strategy not just to defend against data loss/theft but to recover from it.
(2) Understand the "big" picture. Take some time to understand just how insecure digital data now is for all of us, including journalists, businesses, corporations, nation-states. Read James Bamford's "The Shadow Government" or Misha Glennys "McMafia"
(3) Have a reasonable picture of your enemies and how determined they would be to find your content or stop you from owning it.
(4) Remember the famous (and ancient) network adminstrators maxim: "There are only two kinds of computer users: Those who have lost data and those who will." Always back-up your work. Always work with a net.
(5) Lower your "personal attack surface". Two separate strategies come to mind:
 (a) Confuse possible threats through secrecy, security, iconoclastic behavior, obfuscation and misdirection. (e.g. Keep a 'cover' or 'alibi' or 'grey' lifestyle, own many small computers, own multiple phones but don't always carry them, take public transport to busy malls to work, cultivate unpredictable behavior patterns etc. )
 (b) Become involved and well-known in your community and tribe: develop friends, watchers, and confidants. Keep a respected public content profile on a Blog. Attend your block watch, neighborhood meetings, have your neighbors over for dinner etc.
These two strategies may be more compatible than apparent...
(6) If something feels wrong to you, it probably is. If you don't feel like filling out some Facebook survey that asks for the "top twenty things people don't know about you" your life may well be more secure. Hackers often make up password lists of details from peoples personal lives.  Ask your medical professional exactly why he needs your Social Security number on that form. Despite recent HIPPA laws, medical information is notoriously insecure. The list goes on: too personal strangers, tele-funders from not well-known organizations asking for your credit card numbers.  Limit the leakage of critical personal information. Often, no one else needs to know. Resist the urge to converse too personal details to strangers.
(7) Don't underestimate the threats. But don't spend too much time worrying about them either. It is well-known that successful personal security always involves intuition and spontaneity. Both are dimmed by too much concern.
  
Part II Safe Computing Practices
As for generally accepted computing security practices, if I had to protect sensitive content I would:
(1) read any number of sites that give excellent recommendations on "safe computing practices" from the NSA to FBI to CERT to SANS to SLATE and USE THEM.
(2) understand my firewall, anti-virus, security templates and encryption suites well and USE THEM.
(3) understand my Operating System/Application suite really well. Monitor Operating System/Application security flaws and update as prescribed.
(4) review my firewall, syslog, eventviewer, anti-virus, and web logs every week and attempt to profile both my audience and possible threats from collating all log information.
(5) use product vendors I believe in for my OS, Applications, firewall, AV, encryption suites. Consider using "open-source" platforms and applications whose code is well-reviewed.
(6) if possible or practical, I would store a non-digital copy of protected content in multiple safe locations to protect from disaster.
(7) not keep secure information or sensitive content in your e-mail. Most e-mail is exchanged in 'clear text' across the wire. Most e-mail stores are not kept in 'data vaults' although some e-mail software will offer you this option.
(8) try to remember that data-loss is multi-faceted and often physical in nature. More data may be lost from stolen laptops in America than through network intrusions: Buy a vault and USE IT. Buy multiple deadbolts and USE THEM. Be careful with your laptops and portable drives when you are in a crowd or public place. (See Part IV, suggestion (2) below.)

True Story: I once heard a friend of mine discuss how the local PD called upon to help him break the encryption on a drive of an uncommon real-time UNIX OS that belonged to a narcotics trafficker.
The dealer had gone to some lengths to use a rare OS with encryption of which few people would have technical experience. But once the local PD had physical control of the box...game over....

Part III More Computer Security Practices
Some more computer security suggestions for content protection:
(1) Receive e-mail in plain text always. Consider sending digitally signed e-mail. 
(2) Encrypt your laptop and hard drives with third party encryption.
(3) Understand file and logon security for your Operating System and deploy and use them.
(4) Deploy host and network firewalls and a honeypot. Consider firewalling different segment of your network. Lately, I like the concept of these new UTM (Unified Threat Modeling) firewalls from NetGear and other vendors...
(5) Learn to sniff and review traffic everywhere you go. There's something satisfying about actually reviewing network traffic as you work on the network. Something like surveying the crowd on the street you are walking on...
(6) Consider carrying a small portable hardware firewall for your laptop.
(7) Get in the practice of quickly reformatting an up to date version of your OS if you feel the least bit quesy about your OS behavior. [This tip implies an excellent data back-up habit and some patience with OS installation.].
(8) Wireless is still a risk, especially if unencrypted. If you use it, use a VPN or encryption for sensitive communication and the highest strength encryption you can afford. Beware of "rogue" public hot spots that steal information.
(9) Use OpenSSH for your network communication as much as possible, especially across networks you don't control or own.
(10) Get in the habit of using 21 character plus passwords and changing them often. Yes, you can.
(11) Regardless of whether you run Windows or UNIX, don't take unneccesary sharing or open port or remote administration risks. 'Lock down' the most expensive version of Windows you can afford (e.g. Vista Ultimate). 
[Note: Securing Windows or UNIX requires some effort and thought and training. The use of a consultant may be advisable.]

Part IV Offbeat Ideas?
On the "inventive" or "offbeat" side, if I had to protect sensitive content I might:
(1) Store everything on an old, cheap OpenBSD box that never touches the internet. Run only OpenBSD approved packages.
(2) Buy a Nokia 810, and install and configure iptables. Use it to Skype with your friends off some random wireless connection instead using a cell-phone. Carry it in your jacket pocket and use it instead of a laptop.
(3) Use an iconoclastic Linux distro designed for security - 'Back-Trax' comes to mind...
(4) Surf and collect e-mail on a thumb drive from a bootable Linux distro. Then boot back into an OpenBSD, Linux, Debian,Ubuntu laptop that has no networking at all for your "protected content".
(5) Or you could do the converse: Surf on your hard drive box, boot into a Linux DVD distro, mount a "secure" thumb drive or SD drive for storing sensitive content.
(6) Keep two sets of content: One that can be "found" by your enemies (after some work) and one that is "hidden".  For example,thumb-drives are easy to purchase, back up, and/or store on your person. You could leave decoys lying around with "disinformative content" for when the spooks do a "sneak and peek" at your apartment.
(7) Set up a surveillance system around your home.
(8) Teach yourself to hack and spy. Nothing will make you more paranoid and careful than knowing the "arts of the enemy". Actually, nothing will intimidate your enemies more than aggressive "back-tracking" of their hacking and spying attempts. 
(9) Live in the most populous neighborhood you can stand. San Francisco's China Town comes to mind. It's hard to follow one person consistently in a crowd.
(10) Publish some of your most problematic secret content on a blog, similar to what the Electronic Freedom Foundation does every month. Nothing makes sensitive content less so than publicity. Sometimes, nothing makes a holding a secret less dangerous than sharing it. 

AND THE NUMBER ONE WAY TO PROTECT SENSITIVE CONTENT IS...
[Live without any. Wasn't life simpler without computers? :-)] 

Parsing Vista Firewall: Part IV

Microsoft's logparser.exe use sql query syntax to parse many different log formats.  Vista's firewall most reasonably resembles at TSV log file format. However, it takes some work with logparser.exe to get the correct parameters as below.  The third or 'header' line row needs  the words "#Fields" removed from the file for accurate field recognition.

LogParser "SELECT * FROM 'pfirewall.log' WHERE ( action = 'ALLOW' AND protocol = 'UDP' AND path = 'RECEIVE' AND src-ip <> '127.0.0.1' ) " -i:TSV -iSeparator:spaces -fixedSep:OFF -nSkipLines:3

Filename RowNumber date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info path
--------------------------------------------------- --------- ---------- -------- ------ -------- --------------- --------------- -------- -------- ---- -------- ------ ------ ------ -------- -------- ---- -------
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 7105 2009-07-11 19:56:59 ALLOW UDP 192.168.0.4 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 7107 2009-07-11 19:56:59 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8046 2009-07-11 21:56:36 ALLOW UDP 192.168.0.4 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8047 2009-07-11 21:56:36 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8316 2009-07-11 22:03:29 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8353 2009-07-11 22:06:18 ALLOW UDP 192.168.0.4 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8355 2009-07-11 22:06:18 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
....

Parsing Vista Firewall Logs: Part I

These are the fields Vista HP logs for C:\Windows\System32\LogFiles\Firewall\pfirewall.log:

#Fields: date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info path

The Log meanders along like as below. Note the IPv6 broadcasts:

...

2009-07-22 08:15:00 ALLOW TCP 192.168.0.11 74.125.95.139 53218 80 0 - 0 0 0 - - - SEND

2009-07-22 08:25:21 ALLOW UDP 192.168.0.8 192.168.0.255 137 137 0 - - - - - - - RECEIVE

2009-07-22 08:25:21 ALLOW UDP 192.168.0.8 192.168.0.255 137 137 0 - - - - - - - RECEIVE

2009-07-22 08:25:31 ALLOW UDP 192.168.0.8 192.168.0.255 138 138 0 - - - - - - - RECEIVE

2009-07-22 08:26:20 ALLOW UDP ::1 ::1 62537 62537 0 - - - - - - - SEND

2009-07-22 08:26:20 DROP UDP 192.168.0.11 192.168.0.1 65300 53 0 - - - - - - - SEND

2009-07-22 08:28:15 ALLOW UDP ::1 ff02::c 54218 3702 0 - - - - - - - SEND

2009-07-22 08:28:15 ALLOW UDP ::1 ff02::c 54218 3702 0 - - - - - - - RECEIVE

2009-07-22 08:28:15 ALLOW UDP ::1 ff02::c 54218 3702 0 - - - - - - - RECEIVE

2009-07-22 08:28:15 ALLOW UDP fe80::2c20:349c:3f57:fff4 ff02::c 54218 3702 0 - - - - - - - SEND

2009-07-22 08:28:47 DROP UDP 192.168.0.11 192.168.0.1 52197 53 0 - - - - - - - SEND

...

Without pcregrep, grep, gawk, awk, uniq, [unix] sort, gnuplot, etc...parsing is problematic from the native windows cmd shell. The batch below works some magic, but we will need logparser.exe and/or powershell (or GNUWin32 or Cygwin) to do better faster parsing magic:

[ParseIPAllowSort.cmd]

findstr ALLOW pfirewall.log > Allowed.txt

for /f "tokens=1-8" %%a in (Allowed.txt) do @echo %%f %%h ^<^- %%e %%g %%a %%b >> AllowIP.txt

sort /r AllowIP.txt > SortAllowIP.txt

[output]

...

99.31.167.59 57604 <- 192.168.0.11 28656 2009-07-20 17:41:09

99.247.53.140 53591 <- 192.168.0.11 28656 2009-07-21 20:10:00

99.245.94.5 23791 <- 192.168.0.11 28656 2009-07-21 20:10:00

99.239.214.107 39950 <- 192.168.0.11 28656 2009-07-21 20:10:00

99.235.142.40 50446 <- 192.168.0.11 28656 2009-07-21 20:10:00

99.233.190.117 443 <- 192.168.0.11 28656 2009-07-21 20:10:00

99.172.37.127 38445 <- 192.168.0.11 28656 2009-07-20 18:21:45

98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-21 20:10:00

98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-20 20:44:03

98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-20 19:23:52

98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-20 18:21:27

98.28.188.233 25431 <- 192.168.0.11 28656 2009-07-20 17:48:08

98.249.81.221 10969 <- 192.168.0.11 28656 2009-07-21 08:46:43

98.249.81.221 10969 <- 192.168.0.11 28656 2009-07-20 21:39:58

.....

If we launch a scan against my host:

nmap -p 1-65535 -PN ScanTarget

Windump.exe with the latest Winpcap driver is very busy trying to log evey attempt:

C:\Users\admin\Documents\Downloads>windump -vvveXX -s 0 -i 1

windump: listening on \Device\NPF_{0F82FB9D-391A-4293-9D7A-215F53E05FAE}

21:59:20.361046 00:0f:b0:fd:44:2d (oui Unknown) > 00:1d:ba:8a:dc:28 (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 41, id 3782, off

set 0, flags [none], proto: TCP (6), length: 44) ScanHost.36950 > ScanTarget.11165: S, cksum 0x2072 (correct), 4211909807:4211909807(0) win 2048

0>

0x0000: 001d ba8a dc28 000f b0fd 442d 0800 4500 .....(....D-..E.

0x0010: 002c 0ec6 0000 2906 6f02 0a00 0003 0a00 .,....).o.......

0x0020: 0002 9056 2b9d fb0c a4af 0000 0000 6002 ...V+.........`.

0x0030: 0800 2072 0000 0204 05b4 0000 ...r........

.....

Vista Firewall apparently logs some attempts (perhaps enough to show a scanning pattern) and then drops the rest from the log. The packets kept look like this:

...

2009-07-21 21:48:12 DROP TCP 10.0.0.3 10.0.0.2 36950 445 44 S 4211909807 0 2048 - - - RECEIVE

2009-07-21 21:48:12 DROP TCP 10.0.0.3 10.0.0.2 36950 139 44 S 4211909807 0 3072 - - - RECEIVE

2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36951 139 44 S 4211975342 0 4096 - - - RECEIVE

2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36951 445 44 S 4211975342 0 4096 - - - RECEIVE

2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36950 135 44 S 4211909807 0 3072 - - - RECEIVE

2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36951 135 44 S 4211975342 0 4096 - - - RECEIVE

....