Historic blog. No longer active. See Also http://horizontal-logic.blogspot.com for more Powershell code. AS of 2/27/2014 all Scripts are PS 4.0.
Sunday, March 16, 2014
Avoiding XPath: Part VI
So this piece will begin a discussion about grokking Windows Security Event Logs inside rdata.table and Postgres. Use auditpol to set up kernel logging. We recall that to convert your EVTX archived security logs to CSV we need a Powershell function as below:
# Powershell Memory and CPU intensive
Function Convert-Logs3 {
[cmdletbinding()]
Param(
$filelist=$NULL
)
$filelist | foreach-object {
#Note that I am only getting four columns of data
Get-WinEvent -Path "$PSItem"| Select RecordID,ID,TimeCreated, Message | export-csv -notypeinformation -path $(write "$PSItem.csv");
[System.gc]::collect();
}
}
Convert-Logs3 "Archive-Security-2019-12-23-04-25-01-062.evtx"
#R
library(data.table)
d <- fread("Archive-Security-2019-12-23-04-25-01-062.evtx.csv")
h <- gsub('\n\t',' ',d$Message,fixed=TRUE)
h <- gsub('\n\n',' ',h,fixed=TRUE)
h <- gsub('\t\t',' ',h,fixed=TRUE)
h <- gsub('\n',' ',h,fixed=TRUE)
h <- gsub('\t',' ',h,fixed=TRUE)
h <- gsub('%%','',h,fixed=TRUE)
h <- gsub('\r','',h,fixed=TRUE)
h <- gsub('\r \r','',h,fixed=TRUE)
d$Message <- h
This produces row headings with data types:
names(d)
[1] "RecordId" "Id" "TimeCreated" "Message"
d[,sapply(.SD,class)]
RecordId Id TimeCreated Message
"integer" "integer" "character" "character"
# where you can use sample rdata.table queries like this:
d[,.N,.(Id,Message=substr(Message,0,100))][order(-N)]
d[Id == 4688,.(Id,TimeCreated,Message=substr(Message,275,375))]
d[Id == 4672 & !duplicated(substr(TimeCreated,0,11)),
.(Id,TimeCreated,Date=substr(TimeCreated,0,11),Message=substr(Message,400,500))]
d[Id == 4624 & grepl("S-1-0-0",Message),.(Id,Message=substr(Message,200,400))]
d[Id == 4907,.(Id,TimeCreated,paste0(substr(Message,0,50),substr(Message,250,375)))]
There is probably much more you can do with Security.evtx files with mlr3 or rdata.table
# and then...I write out the file to be imported into Postgres:
write-csv(d, "Archive-Security-2019-12-23-04-25-01-062.csv")
Sunday, April 18, 2010
tcpslice II
More uses for tcpslice, ipsumdump, BASH 4.1 :
[This gives you today's top source IP and source IP Port combination:
/usr/sbin/tcpslice `date +%Y"y"%m"m"%d"d"` $BASH_ARGV | ipsumdump --no-headers -sD -
./todays_dump.sh MarApr.snort.in.tcpd | sort -nr | uniq -c | sort -nr
13 85.144.201.237 7959
3 95.179.99.147 5900
3 64.206.157.2 23
3 222.45.112.59 8085
3 109.187.8.70 5900
2 98.247.214.152 23 ...
This gives you today's top source IP and source IP location:
/usr/sbin/tcpslice `date +%Y"y"%m"m"%d"d"` $BASH_ARGV |
for i in `ipsumdump --no-headers -s -`
do echo $i : $(printf "%s" `./geoip.sh $i | awk -F":" '{print $2}' | awk -F"," '{print $1","$2","$3}' ` )
done
./tgeodump.sh MarApr.snort.in.tcpd | sort -nr | uniq -c | sort -nr
13 85.144.201.237 : NL,07,Amsterdam
12 222.45.112.59 : CN,22,Beijing
4 222.215.230.49 : CN,32,Chengdu
3 95.179.99.147 : RU,43,Lipetsk
3 64.206.157.2 : US,NH,Nashua
3 109.187.8.70 : IPAddressnotfound,,
2 98.247.214.152 : US,WA,Bothell ...
where 'geoip.sh' is:
geoiplookup -f /usr/local/share/GeoIP/GeoLiteCity.dat $1
I note that file names like this '08Mar1142PST2010.in.1268074842' don't process through tcpslice.
Wednesday, June 2, 2010
time stamping windows directory and file names
:: rtime.cmd
@echo off
set realdate=%date:/=.%
set realdate=%realdate:* =%
set realtime=%time::=.%
set realtime=%realtime:* =%
set timestamp=%realdate%.%realtime%
echo %timestamp%
This command script uses 'variable substitution' from the set command to remove special characters (e.g. : / ) unacceptable as Windows file or directory names . This line:
set timestamp=%realdate%.%realtime%
can be changed as needed for more CSV compatible logging:
set timestamp="%realdate%","%realtime%"
Once cached, it runs pretty fast and is suitable for lightweight logging:
$ time /cygdrive/C/Security/rtime.cmd
06.02.2010.11.04.05.99
real 0m0.202s
user 0m0.015s
sys 0m0.031s
$ time /cygdrive/C/Security/rtime.cmd
06.02.2010.11.04.12.65
real 0m0.062s
user 0m0.000s
sys 0m0.015s
$ time /cygdrive/C/Security/rtime.cmd
06.02.2010.11.04.14.68
real 0m0.062s
user 0m0.000s
sys 0m0.015s
Tuesday, May 25, 2010
piping tcpdump output to lsof
[Set to the interface of your choice]
while [ 1 ]
do
for i in `tcpdump -i rl0 -c 1 -l dst $(hostname) | awk '{print $2}' | awk -F"." '{print $1"."$2"."$3"."$4}'`
do lsof -i@$i
done
done
with time/date stamp added and headers removed:
while [ 1 ]
do
for i in `tcpdump -i rl0 -c 1 -l dst $(hostname) | awk '{print $2}' | awk -F"." '{print $1"."$2"."$3"."$4}'`
do echo `date -u` `lsof -i@$i | grep -v PID`
done
done
Run like this:
./tcp_lsof.sh >> tcp.lsof.log &
the script produces output like this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 18392 rferrisx 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:13974 (ESTABLISHED)
sshd 29850 root 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:13974 (ESTABLISHED)
or
Wed May 26 15:22:06 UTC 2010 sshd 9448 root 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:15729 (ESTABLISHED)
sshd 29734 rferrisx 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:15729 (ESTABLISHED)
Wed May 26 15:22:07 UTC 2010 sshd 9448 root 5u IPv4 0xd699ac80 0t0 TCP rmflaptop.rmfdevelopment.com:ssh->192.168.0.3:15729 (ESTABLISHED)
sshd 29734 rferrisx 5u IPv4 0xd699ac80
Thursday, February 27, 2014
Avoiding XPath: Part IV
Message : The Windows Filtering Platform has permitted a connection.
Application Information:
Process ID: 3116
Application Name: \device\harddiskvolume3\users\rferrisx\appdata\local\chromium\application\chrome.exe
Network Information:
Direction: Outbound
Source Address: 192.168.0.11
Source Port: 2094
Destination Address: 8.247.65.200
Destination Port: 80
Protocol: 6
Filter Information:
Filter Run-Time ID: 211332
Layer Name: Connect
Layer Run-Time ID: 48
So I can get to these fields with 'properties':
Thursday, July 14, 2011
Get-WinEvent, EventLogs, ETL, Providers on Win7 Part III
Tuesday, July 5, 2011
Get-Winevent Part III: Querying the Event Log for Logons (Part D)
Sunday, January 23, 2011
Get-WinEvent, EventLogs, ETL, Providers on Win7
'Get-WinEvent' in Powerhsell 2 when combined with ETL on Windows 7 allows exceptional event log queries. This function allows the administrator to create an array of all Event Logs and sort by 'time created' all those records created in the last (1) day:
function global:LatestLogEntries
{
[CmdletBinding()]
Param(
[Parameter(Mandatory=$true,ValueFromPipeline=$true)]
[int32] $param1,
[string] $ErrorActionPreference="silentlycontinue"
)
$LogNames=(Get-Winevent -listlog * )
$goback = (get-date) - (new-timespan -days $param1 )
$LogNames | % {get-winevent -FilterHashTable @{LogName=$_.LogName;StartTime=$goback}}
}
Wednesday, July 22, 2009
Parsing Vista Firewall Logs: Part I
Monday, April 30, 2012
Get-Winevent Part IV: Querying the Event Log for 'Filtering Platform Connection' Information (Part A)
The command:
'auditpol /set /subcategory:"Filtering Platform Connection" /success:enable /failure:enable'
enables the "Filtering Platform Connection" security counter on Windows 7. The "Filtering Platform Connection" gives your event logs access to the following counters:
Filtering Platform Connection Success and Failure
- Object Access Filtering Platform Connection 5150 The Windows Filtering Platform has blocked a packet. Windows 7, Windows Server 2008 R2
- Object Access Filtering Platform Connection 5151 A more restrictive Windows Filtering Platform filter has blocked a packet. Windows 7, Windows Server 2008 R2
- Object Access Filtering Platform Packet Drop 5152 The Windows Filtering Platform blocked a packet. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Packet Drop 5153 A more restrictive Windows Filtering Platform filter has blocked a packet. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Connection 5154 The Windows Filtering Platform has permitted an application or service to listen on a port for incoming connections. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Connection 5155 The Windows Filtering Platform has blocked an application or service from listening on a port for incoming connections. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Connection 5156 The Windows Filtering Platform has allowed a connection. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Connection 5157 The Windows Filtering Platform has blocked a connection. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Connection 5158 The Windows Filtering Platform has permitted a bind to a local port. Windows Vista, Windows Server 2008
- Object Access Filtering Platform Connection 5159 The Windows Filtering Platform has blocked a bind to a local port. Windows Vista, Windows Server 2008
[array]$a=Get-WinEvent -FilterHashTable @{LogName='Security';ID=5156;StartTime=$StartTime}
$UDA_count=$a.count
[array[]]$b=$a.Message | findstr 'Destination' | findstr 'Address'
$Global:UDestAddress=($b | Select -unique) | sort
The script takes an extremely long time to run on my five core laptop. These scripts (1,2) are optimized a bit more to search for only 5156 Events. The global variables in the script would be suitable for parsing against lists of allowed ports, allowed or blocked IPs. The Script can be used as a format for other counters as well. Several features from Powershell 3.0 are used in this script including the ability of Powershell 3.0 to 'automatically unroll' an entire array for a certain property (e.g. '[array[]]$b=$a.Message'). I could dearly use a much faster Powershell method to dig 'subfield' information out of the Message field than double piping that information to 'findstr'. The issue is that a single day of network activity generates ten of thousands of kernel security counters. An alternative to limit the amount of information returned might be to use the '-max' [number of events] parameter:
Wednesday, May 13, 2009
Understanding an attack
/usr/local/bin/snort -devX -i xl0 -L $(date "+%b%e%H%M%S%Z%Y") 'port not(domain or whois or http or https or syslog or ntp or smtp or 137 or 139)' and 'not(broadcast or icmp or igmp or arp)'
After some awkward awk statements and some ditzy KSH work, we have a list of ports others who are seeking our network seem interested in:
snort -vdeX -r May12085154PDT2009.1242143514 | grep TTL: | awk -F"->" '{print $1 ":" $2 ":" $3}' | awk -F":" '{print $4}' | awk -F" " '{print $1}' | sort -nr | uniq -c | sort -nr
14 2967
6 8000
6 5900
6 3128
6 22
5 23
5 1434
5 12712
4 7212
4 4899
4 1433
1 8080
1 7209
1 65535
1 56017
1 3306
1 23803
1 21
1 19756
1 19696
1 1024
snort -vdeX -r May12085154PDT2009.1242143514 | grep TTL: | awk -F"->" '{print $1 ":" $2 ":" $3}' | awk -F":" '{print $4}' | awk -F" " '{print $1}' | sort | uniq | sort -nr >> ports.txt
for i in `cat ports.txt`; do grep -w $i /usr/local/share/nmap/nmap-services;done
http-proxy 8080/tcp # Common HTTP proxy/second web server port
http-alt 8000/tcp # A common alternative http port
vnc 5900/tcp # Virtual Network Computer display
radmin 4899/tcp # Radmin (www.radmin.com) remote PC control software
mysql 3306/tcp # mySQL
squid-http 3128/tcp #
symantec-av 2967/udp # Symantec AntiVirus (rtvscan.exe)
ms-sql-m 1434/tcp # Microsoft-SQL-Monitor
ms-sql-m 1434/udp # Microsoft-SQL-Monitor
ms-sql-s 1433/tcp # Microsoft-SQL-Server
ms-sql-s 1433/udp # Microsoft-SQL-Server
kdm 1024/tcp # K Display Manager (KDE version of xdm)
telnet 23/tcp #
telnet 23/udp #
ssh 22/tcp # Secure Shell Login
ssh 22/udp # Secure Shell Login
ftp 21/tcp # File Transfer [Control]
ftp 21/udp # File Transfer [Control]
Nmap services file helps explain much here, but why the large interest in a Symantec AntiVirus port? It turns out others have noticed this recently as well and are asking for input:
http://isc.sans.org/diary.html?storyid=6319.
http://msmvps.com/blogs/harrywaldron/archive/2006/11/27/new-botnet-impacts-symantec-client-port-2967-on-unpatched-pcs.aspx
http://www.offensivecomputing.net/?q=node/403
Is this a new or mutated trojan? worm? remote exploit? Multiple addresses are interested in connecting to us on this port:
# snort -vdeX -r May12085154PDT2009.1242143514 | grep TTL: | grep 2967 | sort -nr | uniq -c | sort -nr
3 218.75.95.242:6000 -> 192.168.0.12:2967 TCP TTL:105 TOS:0x20 ID:256 IpLen:20 DgmLen:40
2 119.161.130.75:6000 -> 192.168.0.12:2967 TCP TTL:99 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 61.191.63.8:6000 -> 192.168.0.12:2967 TCP TTL:103 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 61.145.62.75:6000 -> 192.168.0.12:2967 TCP TTL:107 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 60.173.12.60:6000 -> 192.168.0.12:2967 TCP TTL:106 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 60.172.229.11:6000 -> 192.168.0.12:2967 TCP TTL:105 TOS:0x20 ID:65419 IpLen:20 DgmLen:40
1 60.172.229.11:6000 -> 192.168.0.12:2967 TCP TTL:105 TOS:0x20 ID:42349 IpLen:20 DgmLen:40
1 222.186.26.93:6000 -> 192.168.0.12:2967 TCP TTL:103 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 121.140.174.105:6000 -> 192.168.0.12:2967 TCP TTL:107 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 121.14.156.149:6000 -> 192.168.0.12:2967 TCP TTL:102 TOS:0x20 ID:256 IpLen:20 DgmLen:40
1 121.14.156.148:6000 -> 192.168.0.12:2967 TCP TTL:103 TOS:0x20 ID:256 IpLen:20 DgmLen:40
But the packet looks like a simple connection attempt to a remote port. A buffer overflow in Symantec AntiVirus port?
218.75.95.242:6000 -> 192.168.0.12:2967 TCP TTL:105 TOS:0x20 ID:256 IpLen:20 DgmLen:40
******S* Seq: 0x60B40000 Ack: 0x0 Win: 0x4000 TcpLen: 20
0x0000: 00 60 97 30 6B C4 00 09 5B 00 F3 DA 08 00 45 20 .`.0k...[.....E
0x0010: 00 28 01 00 00 00 69 06 55 BE DA 4B 5F F2 C0 A8 .(....i.U..K_...
0x0020: 00 0C 17 70 0B 97 60 B4 00 00 00 00 00 00 50 02 ...p..`.......P.
0x0030: 40 00 F1 34 00 00 00 00 00 00 00 00 @..4........
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
05/13-12:59:50.507559 0:9:5B:0:F3:DA -> 0:60:97:30:6B:C4 type:0x800 len:0x3C
121.140.174.105:6000 -> 192.168.0.12:2967 TCP TTL:107 TOS:0x20 ID:256 IpLen:20 DgmLen:40
******S* Seq: 0x4260000 Ack: 0x0 Win: 0x4000 TcpLen: 20
0x0000: 00 60 97 30 6B C4 00 09 5B 00 F3 DA 08 00 45 20 .`.0k...[.....E
0x0010: 00 28 01 00 00 00 6B 06 66 06 79 8C AE 69 C0 A8 .(....k.f.y..i..
0x0020: 00 0C 17 70 0B 97 04 26 00 00 00 00 00 00 50 02 ...p...&......P.
0x0030: 40 00 60 0B 00 00 00 00 00 00 00 00 @.`.........
Sunday, May 31, 2009
The National Cyber Security Effort
Wednesday, August 31, 2011
Muxing AccessControl and FileInfo objects
- PSChildName NoteProperty System.String PSChildName=test.txt
- PSDrive NoteProperty System.Management.Automation.PSDriveInfo PSDrive=C
- PSParentPath NoteProperty System.String PSParentPath=Microsoft.PowerShell.Core\FileSystem::C:\
- PSPath NoteProperty System.String PSPath=Microsoft.PowerShell.Core\FileSystem::C:\test.txt
- PSProvider NoteProperty
Friday, June 5, 2009
PolyMorphic Multifunctional Spyware/Malware/Distribution Ware...
Saturday, April 25, 2009
LinuxFest Northwest 2009
# Checks NetGear Firewall syslog output or tcpdump -vvveX of syslog port thereof
# This Bash shell script needs full path to log file as its only args
# Apr24213621PDT2009 rferrisx
newdir=`date "+%b%e%H%M%S%Z%Y"`
mkdir $newdir
cd $newdir
echo City: > whois.search
echo descr: >> whois.search
echo NET >> whois.search
echo owner: >> whois.search
echo NetName >> whois.search
echo Copyright >> whois.search
logfile=$BASH_ARGV
grep -A 1 "Severity alert" $logfile | grep "Msg:" > ./attackers.txt
awk -F":" '{ print $4 "," $6 $7 }' ./attackers.txt > ./SipDipPortDesc.txt
SIP=`awk -F"," '{ print $1 }' ./SipDipPortDesc.txt`
DIP=`awk -F"," '{ print $2 }' ./SipDipPortDesc.txt`
PORT=`awk -F" " '{ print $2 }' ./SipDipPortDesc.txt`
DESC=`awk -F" " '{ print $3 "," $4 }' ./SipDipPortDesc.txt`
WHOIS=`for i in $SIP; do whois $i | grep -m 1 -f whois.search;done;`
echo "$SIP" > SIP
echo "$DIP" > DIP
echo "$PORT" > PORT
echo "$DESC" > DESC
echo "$WHOIS" > WHOIS
for i in $SIP;do traceroute -I -d $i;done > ICMPtraceroute.txt
Monday, May 4, 2009
Monitoring home networks with bare bones
Saturday, August 1, 2009
Parsing Vista Firewall: Part IV
Microsoft's logparser.exe use sql query syntax to parse many different log formats. Vista's firewall most reasonably resembles at TSV log file format. However, it takes some work with logparser.exe to get the correct parameters as below. The third or 'header' line row needs the words "#Fields" removed from the file for accurate field recognition.
LogParser "SELECT * FROM 'pfirewall.log' WHERE ( action = 'ALLOW' AND protocol = 'UDP' AND path = 'RECEIVE' AND src-ip <> '127.0.0.1' ) " -i:TSV -iSeparator:spaces -fixedSep:OFF -nSkipLines:3
Filename RowNumber date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info path
--------------------------------------------------- --------- ---------- -------- ------ -------- --------------- --------------- -------- -------- ---- -------- ------ ------ ------ -------- -------- ---- -------
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 7105 2009-07-11 19:56:59 ALLOW UDP 192.168.0.4 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 7107 2009-07-11 19:56:59 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8046 2009-07-11 21:56:36 ALLOW UDP 192.168.0.4 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8047 2009-07-11 21:56:36 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8316 2009-07-11 22:03:29 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8353 2009-07-11 22:06:18 ALLOW UDP 192.168.0.4 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
C:\Program Files (x86)\Log Parser 2.2\pfirewall.log 8355 2009-07-11 22:06:18 ALLOW UDP 169.254.172.113 239.255.255.250 51493 1900 0 - - - - - - - RECEIVE
....
Tuesday, February 9, 2010
Advanced Persistent Threat
(1) Our current desktop and server Operating Systems are not secure.
(2) Computer networks are insecure for most organizations and at many levels.
(3) Digital data can no longer be protected against a determined foe.
(4) Security researchers and visionaries should receive more funding. Lots.
Order and read the Mandiant Report. Then imagine what a resourced foe could do if they believed the security of their nation-state depended upon seemless corporate intrusions. Now imagine those techniques automated and in the wild. In order for the world to have safe computing systems, our government and industry needs to sponsor more research and decriminalize vulnerability research. Otherwise, no data will ever be secret or protected again.