For speed, control, and simplicity, gawk is almost impossible to beat in parsing simple text logs like pfirewall.log. The script below will give you a numerically sorted list by count of the references to Src IPs in pfirewall.log for allowed packets. These sorts give a count (first column) of the unique IPs in numerical order. Note that gawk makes quick work of this searches.
gawk '$3 == "ALLOW" {print $5}' pfirewall.log | sort -nr | uniq -c | sort -nr
6849 192.168.0.4
4317 127.0.0.1
3014 192.168.200.87
1577 10.10.10.74
725 192.168.168.246
680 172.17.5.143
595 fe80::9536:4516:f99:3705
557 ::1
350 fe80::645d:d71d:f845:ac71
265 192.168.150.10
261 169.254.172.113
214 0.0.0.0
122 10.10.10.82
107 85.13.200.108
...
Now we add the Src IP ports:
gawk '$3 == "ALLOW" {print $5" "$7}' pfirewall.log | sort -nr | uniq -c | sort -nr
1609 127.0.0.1 58915
1341 127.0.0.1 58912
214 0.0.0.0 68
132 fe80::9536:4516:f99:3705 -
128 192.168.0.4 137
116 fe80::645d:d71d:f845:ac71 -
107 85.13.200.108 20
106 ::1 -
106 127.0.0.1 -
96 127.0.0.1 52845
76 fe80::ffff:ffff:fffe -
73 127.0.0.1 53249
72 169.254.172.113 137
....
Now we add the DestIP and Dest Ports:
gawk '$3 == "ALLOW" {print $5" "$6" "$8}' pfirewall.log | sort -nr | uniq -c | sort -nr
1609 127.0.0.1 127.0.0.1 58915
1364 192.168.0.4 192.168.0.1 53
1341 127.0.0.1 127.0.0.1 58912
720 192.168.0.4 208.113.141.123 80
668 127.0.0.1 239.255.255.250 1900
661 192.168.200.87 192.168.200.1 53
461 fe80::9536:4516:f99:3705 ff02::1:3 5355
389 10.10.10.74 10.10.10.1 53
379 192.168.0.4 192.168.0.245 80
235 192.168.0.4 69.63.176.175 80
233 fe80::645d:d71d:f845:ac71 ff02::1:3 5355
214 0.0.0.0 255.255.255.255 67
172 192.168.0.4 224.0.0.252 5355
....
Now we sort SrcIP, DestIP, DestPort by uniq IP:
gawk '$3 == "ALLOW" {print $5" "$6" "$8}' pfirewall.log | sort -k 1,3 | uniq -c
214 0.0.0.0 255.255.255.255 67
25 10.0.0.4 10.0.0.255 137
7 10.0.0.4 224.0.0.22 -
1 10.0.0.4 224.0.0.252 137
63 10.0.0.4 224.0.0.252 5355
1 10.0.0.4 239.255.255.250 3702
1 10.10.10.10 224.0.0.1 -
1 10.10.10.74 10.10.10.1 137
13 10.10.10.74 10.10.10.1 2060
389 10.10.10.74 10.10.10.1 53
1 10.10.10.74 10.10.10.1 67
19 10.10.10.74 10.10.10.255 137
2 10.10.10.74 12.129.210.71 80
2 10.10.10.74 12.129.210.76 80
...
As above, but now sorted by count of Uniq IP:
gawk '$3 == "ALLOW" {print $5" "$6" "$8}' pfirewall.log | sort -k 1,3 | uniq -c | sort -nr
1609 127.0.0.1 127.0.0.1 58915
1364 192.168.0.4 192.168.0.1 53
1341 127.0.0.1 127.0.0.1 58912
720 192.168.0.4 208.113.141.123 80
664 127.0.0.1 239.255.255.250 1900
661 192.168.200.87 192.168.200.1 53
461 fe80::9536:4516:f99:3705 ff02::1:3 5355
389 10.10.10.74 10.10.10.1 53
379 192.168.0.4 192.168.0.245 80
235 192.168.0.4 69.63.176.175 80
233 fe80::645d:d71d:f845:ac71 ff02::1:3 5355
214 0.0.0.0 255.255.255.255 67
172 192.168.0.4 224.0.0.252 5355
167 169.254.172.113 224.0.0.252 5355
154 172.17.5.143 172.17.5.1 53
147 192.168.0.4 207.115.66.86 80
140 192.168.150.10 192.168.150.1 53
136 192.168.200.87 206.223.158.41 443
...
Wednesday, July 29, 2009
Tuesday, July 28, 2009
Parsing Vista Firewall Logs Part II
Made an interesting attempt today to parse Vista's Firewall log based on some "Scripting Guys" code from Microsoft: http://www.microsoft.com/technet/scriptcenter/resources/qanda/apr09/hey0416.mspx. I have placed the script here: http://www.rmfdevelopment.com/PowerShell_Scripts/Scan-Firewall.ps1
Regexing per line of pfirewall.log with mixed IPv4 and IPv6 address types as well as ICMP and other layer 2 protocols makes identifying network services by port unreliable without tokenizing the position of dst/src ports first. Thus the regex switch "\s80" will catch web browsing by any local user and "http tunnelling" attempts by any hijacker. Whether or not services correspond to port numbers is a matter of configuration. For example, an alternative web server port is often 8000 as opposed to 80. Regex statistics for [un]cataloged ports may add up to more or less than 100% because of duplicate src/dst ports and/or uncataloged port numbers or alternative network service ports or other pfirewall.log anomalies. Standards based locations for well-known ports exists in the services file of most Operating Systems. On Vista: "C:\Windows\System32\drivers\etc\services". You can use this file to add services to the 'switch -regex' Function and $hash hashtable in the script.
PS C:\PS1> .\Scan-Firewall.ps1
Name Value
---- -----
ssh
ftp
telnet
pop3 3
ntp 4
nbsession 7
microsoftds 43
icmp 643
dhcpc 692.020425632398
ssl 936
ssdp 1223
nbdatagram 3542
llmnr 5440
web 8077
tcp 9222
dns 9417
nbname 12021
PacketAllow 19673
PacketDrop 19999
udp 29576
.
Summary Statistics Layer 2/3 Protocols
.
Total Packets = 39672
Percent Packets Allowed = 0.495891308731599
Percent Packets Dropped = 0.504108691268401
Percent TCP = 0.232456140350877
Percent UDP = 0.745513208308127
Percent ICMP = 0.0162079048195201
Count other Packets = 231
% Other Layer 2/3 Protocols = 0.005822747
.
Summary Statistics IP Application Protocols Per Port
.
Port 67 Percent DHCPC packets = 0.0174435477322141
Port 80 Percent WEB packets = 0.203594474692478
Port 110 Percent POP3 packets = 7.56200846944949E-05
Port 123 Percent NTP packets = 0.00010082677959266
Port 137 Percent NBNAME packets = 0.303009679370841
Port 138 Percent NBDATAGRAM packets = 0.0892821133293003
Port 139 Percent NBSESSION packets = 0.000176446864287155
Port 443 Percent SSL packets = 0.0235934664246824
Port 445 Percent Microsoft DS packets = 0.00108388788062109
Port 1900 Percent SSDP packets = 0.0308277878604557
Port 5355 Percent LLMNR packets = 0.137124420246017
Percent Cataloged Ports = 0.8062366
Percent Uncataloged Ports = 0.1937634
Wednesday, July 22, 2009
Parsing Vista Firewall Logs: Part I
These are the fields Vista HP logs for C:\Windows\System32\LogFiles\Firewall\pfirewall.log:
#Fields: date time action protocol src-ip dst-ip src-port dst-port size tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info path
The Log meanders along like as below. Note the IPv6 broadcasts:
...
2009-07-22 08:15:00 ALLOW TCP 192.168.0.11 74.125.95.139 53218 80 0 - 0 0 0 - - - SEND
2009-07-22 08:25:21 ALLOW UDP 192.168.0.8 192.168.0.255 137 137 0 - - - - - - - RECEIVE
2009-07-22 08:25:21 ALLOW UDP 192.168.0.8 192.168.0.255 137 137 0 - - - - - - - RECEIVE
2009-07-22 08:25:31 ALLOW UDP 192.168.0.8 192.168.0.255 138 138 0 - - - - - - - RECEIVE
2009-07-22 08:26:20 ALLOW UDP ::1 ::1 62537 62537 0 - - - - - - - SEND
2009-07-22 08:26:20 DROP UDP 192.168.0.11 192.168.0.1 65300 53 0 - - - - - - - SEND
2009-07-22 08:28:15 ALLOW UDP ::1 ff02::c 54218 3702 0 - - - - - - - SEND
2009-07-22 08:28:15 ALLOW UDP ::1 ff02::c 54218 3702 0 - - - - - - - RECEIVE
2009-07-22 08:28:15 ALLOW UDP ::1 ff02::c 54218 3702 0 - - - - - - - RECEIVE
2009-07-22 08:28:15 ALLOW UDP fe80::2c20:349c:3f57:fff4 ff02::c 54218 3702 0 - - - - - - - SEND
2009-07-22 08:28:47 DROP UDP 192.168.0.11 192.168.0.1 52197 53 0 - - - - - - - SEND
...
Without pcregrep, grep, gawk, awk, uniq, [unix] sort, gnuplot, etc...parsing is problematic from the native windows cmd shell. The batch below works some magic, but we will need logparser.exe and/or powershell (or GNUWin32 or Cygwin) to do better faster parsing magic:
[ParseIPAllowSort.cmd]
findstr ALLOW pfirewall.log > Allowed.txt
for /f "tokens=1-8" %%a in (Allowed.txt) do @echo %%f %%h ^<^- %%e %%g %%a %%b >> AllowIP.txt
sort /r AllowIP.txt > SortAllowIP.txt
[output]
...
99.31.167.59 57604 <- 192.168.0.11 28656 2009-07-20 17:41:09
99.247.53.140 53591 <- 192.168.0.11 28656 2009-07-21 20:10:00
99.245.94.5 23791 <- 192.168.0.11 28656 2009-07-21 20:10:00
99.239.214.107 39950 <- 192.168.0.11 28656 2009-07-21 20:10:00
99.235.142.40 50446 <- 192.168.0.11 28656 2009-07-21 20:10:00
99.233.190.117 443 <- 192.168.0.11 28656 2009-07-21 20:10:00
99.172.37.127 38445 <- 192.168.0.11 28656 2009-07-20 18:21:45
98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-21 20:10:00
98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-20 20:44:03
98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-20 19:23:52
98.28.36.109 36940 <- 192.168.0.11 28656 2009-07-20 18:21:27
98.28.188.233 25431 <- 192.168.0.11 28656 2009-07-20 17:48:08
98.249.81.221 10969 <- 192.168.0.11 28656 2009-07-21 08:46:43
98.249.81.221 10969 <- 192.168.0.11 28656 2009-07-20 21:39:58
.....
If we launch a scan against my host:
nmap -p 1-65535 -PN ScanTarget
Windump.exe with the latest Winpcap driver is very busy trying to log evey attempt:
C:\Users\admin\Documents\Downloads>windump -vvveXX -s 0 -i 1
windump: listening on \Device\NPF_{0F82FB9D-391A-4293-9D7A-215F53E05FAE}
21:59:20.361046 00:0f:b0:fd:44:2d (oui Unknown) > 00:1d:ba:8a:dc:28 (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 41, id 3782, off
set 0, flags [none], proto: TCP (6), length: 44) ScanHost.36950 > ScanTarget.11165: S, cksum 0x2072 (correct), 4211909807:4211909807(0) win 2048
0>
0x0000: 001d ba8a dc28 000f b0fd 442d 0800 4500 .....(....D-..E.
0x0010: 002c 0ec6 0000 2906 6f02 0a00 0003 0a00 .,....).o.......
0x0020: 0002 9056 2b9d fb0c a4af 0000 0000 6002 ...V+.........`.
0x0030: 0800 2072 0000 0204 05b4 0000 ...r........
.....
Vista Firewall apparently logs some attempts (perhaps enough to show a scanning pattern) and then drops the rest from the log. The packets kept look like this:
...
2009-07-21 21:48:12 DROP TCP 10.0.0.3 10.0.0.2 36950 445 44 S 4211909807 0 2048 - - - RECEIVE
2009-07-21 21:48:12 DROP TCP 10.0.0.3 10.0.0.2 36950 139 44 S 4211909807 0 3072 - - - RECEIVE
2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36951 139 44 S 4211975342 0 4096 - - - RECEIVE
2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36951 445 44 S 4211975342 0 4096 - - - RECEIVE
2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36950 135 44 S 4211909807 0 3072 - - - RECEIVE
2009-07-21 21:48:13 DROP TCP 10.0.0.3 10.0.0.2 36951 135 44 S 4211975342 0 4096 - - - RECEIVE
....
Tuesday, July 21, 2009
No tail.exe on Vista HP....
Vista's decision to ship without a native tail.exe makes monitoring logs difficult without GNUWin32 or Cygwin or some other third party utility. This batch (tail.cmd) helps:
@echo off
set file = %1
:top
choice /T 1 /D Y > NUL
for /f "eol=: tokens=3" %%i in ('find ^/C " " %1') do set lastline=%%i
set /a newlastline=%lastline% - 1
set oldlastline=%newlastline%
:top1
choice /T 1 /D Y > NUL
for /f "eol=: tokens=3" %%i in ('find ^/C " " %1') do set lastline=%%i
set /a newlastline=%lastline% - 1
if %oldlastline%==%newlastline% (goto top1) else more +%newlastline% %1 && goto top
[output]:
C:\Windows\System32\LogFiles\Firewall>tail pfirewall.log
2009-07-21 14:53:25 DROP UDP 192.168.0.10 192.168.0.255 138 138 229 - - - - - - - RECEIVE
2009-07-21 14:54:43 ALLOW UDP 10.0.0.2 10.0.0.255 138 138 0 - - - - - - - SEND
2009-07-21 14:57:16 DROP UDP 192.168.0.11 192.168.0.1 54103 53 0 - - - - - - - SEND
2009-07-21 14:57:18 DROP UDP 192.168.0.11 192.168.0.1 54103 53 0 - - - - - - - SEND
2009-07-21 14:57:30 ALLOW ICMP 192.168.0.11 192.168.0.1 - - 0 - - - - 8 0 - SEND
2009-07-21 14:57:31 ALLOW ICMP 192.168.0.11 192.168.0.1 - - 0 - - - - 8 0 - SEND
2009-07-21 14:57:37 ALLOW ICMP 192.168.0.11 192.168.0.245 - - 0 - - - - 8 0 - SEND
2009-07-21 14:57:38 ALLOW ICMP 192.168.0.11 192.168.0.245 - - 0 - - - - 8 0 - SEND
2009-07-21 14:58:17 ALLOW UDP fe80::11f2:abb3:cf0b:58d8 ff02::1:3 63249 5355 0 - - - - - - - SEND
2009-07-21 14:58:17 ALLOW UDP 10.0.0.2 224.0.0.252 59011 5355 0 - - - - - - - SEND
Monday, July 20, 2009
"The Cloud" exists already....
For anyone who runs a Windows PC, the vaunted "cloud computing" environment already exists. Without most of us realizing it, large collections of computer systems - CDNs, botnets, grid-enabled NOCs, hosting centers, etc. already provide the type of computing power environments that connect our PCs to a world of search engines/databases/upgrades/virus signatures from many vendors. When my Vista boots it proceeds immediately to find out who feeds it daily medicine and then proceeds to tell the world what state it is in:
Thursday, July 16, 2009
Vista and XP Network Interfaces with Powershell
In preparation for examing the differences between XP and Vista firewalls, I wrote this interesting Powershell CTP2.0 v3 script that exposes info for all physical and virtual network interfaces. More information is available on Vista than XP. This script will work on both. Each paragraph is a separate script.
.\Get-NetworkInterface.ps1
$Global:gwmiw32na = get-wmiobject win32_networkadapter
$gwmiw32na | fl *
$Global:gwmiw32na4 = get-wmiobject win32_networkadapter | Select Name,ServiceName,Speed,PhysicalAdapter
$Global:gwmiw32na6 = get-wmiobject win32_networkadapter | Select Name,PhysicalAdapter,MACAddress,Manufacturer,NetConnectionID
$gwmiw32na4 | ft -auto
$gwmiw32na6 | ft -auto
$gwmiw32na = get-wmiobject win32_networkadapter
$gwmiw32na | ? {$_.PhysicalAdapter} |Select Name,MACAddress,GUID,Description,Manufacturer,NetConnectionID,NetConnectionStatus
$gwmiw32na | ? {$_.PhysicalAdapter}
$adapter = Get-WmiObject win32_networkadapter
$adapter_count = $adapter.count
$adapter_range = ($adapter.count + 1)
write "Adapter Range= $adapter_range; Adapter Count= $adapter_count"
write .
write "Adapter Table:"
$AdapterNameID = $adapter | Select Name,DeviceID
$AdapterNameID
Wednesday, July 1, 2009
Understanding Svchost Part II
I have published a brief papers on svchost.exe: Svchost:To Whom and Why . It explains how to use a Mark Russinovich (Microsoft: www.Sysinternals.com) tool set to understand svchost.exe behavior. Microsoft uses Limelight Networks (among other 'CDNs') to help them distribute update content. What I do not like about this is that when you enable Microsoft update you do not explicitly give Microsoft permission to use a third party CDN to send and receive data from your PC. But that is exactly what happens in the world of Edge Networks, 'CDNs', 'Software Ecosystems' and 'Cloud computing'. Data from my computer is sent elsewhere without my permission to network locations that are not local to the Pacific Northwest or necessarily controlled by the software vendor of which I have service level agreements.
Subscribe to:
Posts (Atom)