Preprocessor Documentation

README.unified2

I. Configuring Unified2 Output

Unified2 can work in one of three modes, packet logging, alert logging, or true unified logging. Packet logging includes a capture of the entire packet and is specified with log_unified2. Likewise, alert logging will only log events and is specified with alert unified2. To include both logging styles in a single, unified file, simply specify unified2.

When MPLS support is turned on, MPLS labels can be included in unified2 events. Use option mpls_event_types to enable this. If option mpls event types is not used, then MPLS labels will be not be included in unified2 events.

  • Note that by default, unified2 files have the create time (in Unix Epoch format) appended to each file when it is created.

Format: output alert_unified2: \ filename [, limit ] [, nostamp] \ [,mpls_event_types] [, vlan_event_types]

output log_unified2: \
    filename <base filename> [, limit <size in MB>] [, nostamp]

output unified2: \
    filename <base filename> [, limit <size in MB>] [, nostamp] \
    [,mpls_event_types] [, vlan_event_types]
  • Note that you’ll need to have compiled snort with –enable-mpls as well as use the mpls_event_types to obtain mpls events.

Example: output alert_unified2: filename snort.alert limit 128, nostamp output log_unified2: filename snort.log, limit 128, nostamp output unified2: filename merged.log, limit 128, nostamp output unified2: filename merged.log, limit 128, nostamp, \ mpls_event_types output unified2: filename merged.log, limit 128, \ mpls_event_types, vlan_event_types

Unified2 also has logging support for various extra data. The following configuration items will enable these extra data logging facilities.

config log_ipv6_extra_data

This option enables Snort to log IPv6 source and destination
address as unified2 extra data events.

enable_xff

This option enables HTTP Inspect to parse and log the original
client IP present in the X-Forwarded-For or True-Client- IP HTTP
request headers along with the generated events.
* see README.http_inspect for more information

log_uri

This option enables HTTP Inspect to parse and log the URI data
from the HTTP request and log it along with all the generated
events for that session.
* see README.http_inspect for more information

log_hostname 

This option enables HTTP Inspect to parse and log the Host header
data from the HTTP request and log it along with all the generated
events for that session.
* see README.http_inspect for more information

log_hostname 

This option enables HTTP Inspect to parse and log the Host header
data from the HTTP request and log it along with all the generated
events for that session.
* see README.http_inspect for more information

log_mailfrom

This option enables SMTP preprocessor to parse and log the senders
email address extracted from the "MAIL FROM" command along with
all the generated events for that session.
* see README.SMTP for more information

log_rcptto

This option enables SMTP preprocessor to parse and log the
recipients email address extracted from the "RCPT FROM" command
along with all the generated events for that session.
* see README.SMTP for more information

log_rcptto

This option enables SMTP preprocessor to parse and log the MIME
attachment filenames extracted from the Content-Disposition header
within the MIME body along with all the generated events for that
session.
* see README.SMTP for more information

log_email_hdrs

This option enables SMTP preprocessor to parse and log the SMTP
email headers extracted from the SMTP data along with all the
generated events for that session.
* see README.SMTP for more information

II. Reading Unified2 Files

A. U2SpewFoo

U2SpewFoo is a lightweight tool for dumping the contents of unified2 files to stdout.

Example usage:

$ u2spewfoo snort.log

Example Output:

(Event)
    sensor id: 0    event id: 4 event second: 1299698138    event microsecond: 146591
    sig id: 1   gen id: 1   revision: 0  classification: 0
    priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
    src port: 60710 dest port: 80   protocol: 6 impact_flag: 0  blocked: 0

Packet
    sensor id: 0    event id: 4 event second: 1299698138
    packet second: 1299698138   packet microsecond: 146591
    linktype: 1 packet_length: 54
[    0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00  ..............E.
[   16] 00 28 00 06 00 00 40 06 5C B7 0A 01 02 03 0A 09  .(....@.\.......
[   32] 08 07 ED 26 00 50 00 00 00 62 00 00 00 2D 50 10  ...&.P...b...-P.
[   48] 01 00 A2 BB 00 00                                ......

(ExtraDataHdr)
    event type: 4   event length: 33

(ExtraData)
    sensor id: 0    event id: 2 event second: 1299698138
    type: 9 datatype: 1 bloblength: 9   HTTP URI: /

(ExtraDataHdr)
    event type: 4   event length: 78

(ExtraData)
    sensor id: 0    event id: 2 event second: 1299698138
    type: 10    datatype: 1 bloblength: 54  HTTP Hostname: србијаицрнагора.иком.museum

B. U2Boat

U2boat is a tool for converting unified2 files into different formats.

Currently supported conversion formats are: pcap

Example usage: u2boat -t pcap

III. Unified2 File Format

Unified 2 records should not be assumed to be in any order. All values are stored in network byte order.

An example structure of unified2 files

[ Serial Unified2 Header    ]
[ Unified2 IDS Event        ]
[ Unified2 Packet           ]
[ Unified2 Extra Data       ]
.
.
.
[ Serial Unified2 Header    ]
[ Unified2 IDS Event        ]
[ Unified2 Packet           ]
[ Unified2 Extra Data       ]

A. Serial Unified2 Header:

record type             4 bytes
record length           4 bytes

All unified2 records are preceded by a Serial Unified2 header. This unified2 record allows an interpreting application to skip past and apply simple heuristics against records.

The Record Type indicates one of the following unified2 records follows the Serial Unified2 Header:

Value           Record Type
----------      -----------
2               Unified2 Packet
7               Unified2 IDS Event
72              Unified2 IDS Event IP6
104             Unified2 IDS Event      (Version 2)
105             Unified2 IDS Event IP6  (Version 2)
110             Unified2 Extra Data

The record length field specifies the entire length of the record (not including the Serial Unified2 Header itself) up to the next Serial Unified2 Header or EOF.

B. Unified2 Packet

sensor id               4 bytes
event id                4 bytes
event seconds           4 bytes
event microseconds      4 bytes
linktype                4 bytes
packet length           4 bytes
packet data             <variable length>

A Unified2 Packet is provided with each Unified2 Event record. This packet is the `alerting’ packet that caused a given event.

Unified2 Packet records contain contain a copy of the packet that caused an alert (Packet Data) and is packet length octets long.

C. Unified2 IDS Event

sensor id               4 bytes
event id                4 bytes
event second            4 bytes
event microsecond       4 bytes
signature id            4 bytes
generator id            4 bytes
signature revision      4 bytes
classification id       4 bytes
priority id             4 bytes
ip source               4 bytes
ip destination          4 bytes
source port/icmp type   2 bytes
dest. port/icmp code    2 bytes
protocol                1 byte
impact flag             1 byte
impact                  1 byte
blocked                 1 byte

Unified2 IDS Event is logged for IPv4 Events without VLAN or MPLS tagging.

D. Unified2 IDS Event IP6

sensor id               4 bytes
event id                4 bytes
event second            4 bytes
event microsecond       4 bytes
signature id            4 bytes
generator id            4 bytes
signature revision      4 bytes
classification id       4 bytes
priority id             4 bytes
ip source               16 bytes
ip destination          16 bytes
source port/icmp type   2 bytes
dest. port/icmp code    2 bytes
protocol                1 byte
impact flag             1 byte
impact                  1 byte
blocked                 1 byte

Unified2 IDS Event IP6 is logged for IPv6 Events without VLAN or MPLS tagging.

E. Unified2 IDS Event (Version 2)

sensor id               4 bytes
event id                4 bytes
event second            4 bytes
event microsecond       4 bytes
signature id            4 bytes
generator id            4 bytes
signature revision      4 bytes
classification id       4 bytes
priority id             4 bytes
ip source               4 bytes
ip destination          4 bytes
source port/icmp type   2 bytes
dest. port/icmp code    2 bytes
protocol                1 byte
impact flag             1 byte
impact                  1 byte
blocked                 1 byte
mpls label              4 bytes
vlan id                 2 bytes
padding                 2 bytes

Unified2 IDS Event (Version 2) are logged for IPv4 packets which contain either MPLS or VLAN headers. Otherwise a Unified2 IDS Event is logged.

  • Note that you’ll need to pass –enable-mpls to configure in order to have Snort fill in the mpls label field.

  • Note that you’ll need to configure unified2 logging with either mpls_event_types or vlan_event_types to get this record type.

F. Unified2 IDS Event IP6 (Version 2)

sensor id               4 bytes
event id                4 bytes
event second            4 bytes
event microsecond       4 bytes
signature id            4 bytes
generator id            4 bytes
signature revision      4 bytes
classification id       4 bytes
priority id             4 bytes
ip source               16 bytes
ip destination          16 bytes
source port/icmp type   2 bytes
dest. port/icmp code    2 bytes
protocol                1 byte
impact flag             1 byte
impact                  1 byte
blocked                 1 byte
mpls label              4 bytes
vlan id                 2 bytes
padding                 2 bytes

Unified2 IDS Event IP6 (Version 2) are logged for IPv6 packets which contain either MPLS or VLAN headers. Otherwise a Unified2 IDS Event IP6 is logged.

  • Note that you’ll need to pass –enable-mpls to configure in order to have Snort fill in the mpls label field.

  • Note that you’ll need to configure unified2 logging with either mpls_event_types or vlan_event_types to get this record type.

G. Unified2 Extra Data

sensor id               4 bytes
event id                4 bytes
event second            4 bytes
type                    4 bytes
data type               4 bytes
data length             4 bytes
data                    <variable length>

H. Description of Fields

Sensor ID Unused

Event ID The upper 2 bytes represent the snort instance, if specified by passing the -G option to Snort.

The lower 2 bytes indicate the unique id of the event.

The Event ID field is used to facilitate the task of coalescing
events with packet data.

Event Seconds and Event Microseconds Timestamp represented as seconds since the epoch of when the alert was generated.

Link Type (Unified2 Packet) The Datalink type of the packet, typically EN10M but could be any of the values as returned by pcap_datalink that Snort handles.

Packet Length (Unified2 Packet) Length of the Packet Data.

Packet Data (Unified2 Packet) The alerting packet, of Packet Length bytes long.

Type (Unified2 Extra Data) Type specifies the type of extra data that was logged, the valid types are:

Value           Description 
----------      -----------
1               Original Client IPv4 
2               Original Client IPv6
3               UNUSED
4               GZIP Decompressed Data
5               SMTP Filename
6               SMTP Mail From
7               SMTP RCPT To
8               SMTP Email Headers
9               HTTP URI
10              HTTP Hostname
11              IPv6 Source Address
12              IPv6 Destination Address
13              Normalized Javascript Data

Data Type (Unified2 Extra Data) The type of extra data in the record.

Value           Description 
----------      -----------
1               Blob  

Data Length (Unified2 Extra Data) Length of the data stored in the extra data record

Data (Unified2 Extra Data) Raw extra event data up to Data Length bytes in size.

All of these Extra data types, with the exception of 1, 2, 11, and
12 (IP Addresses) are stored in plain-text. The IP Address types
need to be interpreted as if they were coming off the wire.

Signature ID The Signature ID of the alerting rule, as specified by the sid keyword.

Generator ID The Generator ID of the alerting rule, as specified by the gid keyword.

Signature Revision Revision of the rule as specified by the rev keyword.

Classification ID Classification ID as mapped in the file classifications.conf

Priority ID Priority of the rule as mapped in the file classifications.conf or overridden by the priority keyword for text rules.

IP Source Source IP of the packet that generated the event.

IP Destination Destination IP of the packet that generated the event.

Source Port/ICMP Type If Protocol is TCP or UDP than this field contains the source port of the alerting packet.

If Protocol is ICMP than this field contains the ICMP type of the
alerting packet.

Destination Port/ICMP Code If protocol is TCP or UDP than this field contains the source port of the alerting packet.

If protocol is icmp than this field contains the icmp code of the
alerting packet.

Protocol Transport protocol of the alerting packet. One of: ip, tcp, udp, or icmp.

Impact flag Legacy field, specifies whether a packet was dropped or not.

Value           Description 
----------      -----------
32              Blocked 

Impact UNUSED; deprecated.

Blocked Whether the packet was not dropped, was dropped or would have been dropped.

Value           Description 
----------      -----------
0               Was NOT Dropped 
1               Was Dropped
2               Would Have Dropped*

* Note that you'll only obtain Would Have Dropped on rules which
  are set to drop while Snort is running in inline-test mode.

MPLS Label (4 bytes) The extracted mpls label from the mpls header in the alerting packet.

VLAN ID The extracted vlan id from the vlan header in the alerting packet.

Padding Padding is used to keep the event structures aligned on a 4 byte boundary.