Avoiding XPath : Part V

(under construction; updated 4/16/2014)

Working with such large files in R has proved to be a real challenge for 4 GB RAM.  I am studying various methods to overcome some of the challeges, but also am considering just working in Postgres or RPostgresSQL.


Once EVTX logs are in CSV format, R has the potential to parse them effectively, although I believe I am one tricky use of 'reshape'  from  doing this in the script below.

Avoiding XPath: Part IV

 Full source  of my unpolished cruft is far below.  If you are going to pull fields out of the Message in Windows Event log without Xpath or XML, how are you going to do it in Powershell 4.0? I will remind you of what the Message field looks like:


Message              : The Windows Filtering Platform has permitted a connection.

                       Application Information:
                           Process ID:        3116
                           Application Name:    \device\harddiskvolume3\users\rferrisx\appdata\local\chromium\application\chrome.exe

                       Network Information:
                           Direction:        Outbound
                           Source Address:        192.168.0.11
                           Source Port:        2094
                           Destination Address:    8.247.65.200
                           Destination Port:        80
                           Protocol:        6

                       Filter Information:
                           Filter Run-Time ID:    211332
                           Layer Name:        Connect
                           Layer Run-Time ID:    48

So I can get to these fields with 'properties':

Avoiding XPath : Part III

(under construction)

Time sequenced lattice chart of  417,834 Windows Security log entries.

This cruft is similar to my post  Avoiding XPath : Part II.  However, here I am (laboriously) converting EVTX to CSV with Powershell 3.0. The files  are sizeable, taking about an hour to convert to CSV. The charts take a non-negligible amount of time to load on my i5 4GB laptop.

Avoiding XPath : Part II

(under construction)

There are a number of issues in understanding Windows security event logging (Auditing). Every 2 - 8 weeks I collect a 614MB Security archive with just my laptop and the audit categories below.  Visualizing and manipulating this data effectively... Here are some solutions using Powershell with R 3.01.

Avoiding XPath: Part I

Typically a Security Event Log entry contains a Message entry that is resistant to anything but an XPath query:

Index         : 65597968
TimeGenerated : 1/29/2014 1:32:01 PM
EventID       : 5158
Message       : The Windows Filtering Platform has permitted a bind to a local port.

                Application Information:
                    Process ID:        200
                    Application Name:    \device\harddiskvolume3\program files (x86)\google\chrome\application\chrome.exe

                Network Information:
                    Source Address:        ::
                    Source Port:        49837
                    Protocol:        17

                Filter Information:
                    Filter Run-Time ID:    0
                    Layer Name:        %%14608
                    Layer Run-Time ID:    38

We can shelve the XPath query by using a combination of Powershell and R to dump the Message field to a single line text entry in a CSV.  Using .NET interfaces to dump the first 1000 Security log entries: