ArcSight Flex Connector Methodology

Version 1.0

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Contents
About Us .......................................................................................................................................... 3 Learning More About Flex Connectors ............................................................................................ 3 Introduction ...................................................................................................................................... 4 Building the Flex Connector ............................................................................................................. 4 Familiarity with Flex Connectors .................................................................................................. 4 Information Gathering ................................................................................................................... 4 Choosing the Flex Connector Type .............................................................................................. 6 Basic Building Blocks ................................................................................................................... 7 Method of Attack .......................................................................................................................... 7 Creating the Parser ....................................................................................................................... 7 Using the FlexAgentWizard .......................................................................................................... 7 Using the Regex Wizard ............................................................................................................... 9 Creating Sub Messages ............................................................................................................... 9 Creating deviceEventClassId ...................................................................................................... 11 Assigning Severity ...................................................................................................................... 11 Creating Map Files...................................................................................................................... 12 Assigning Categorization ............................................................................................................ 12 Testing ........................................................................................................................................ 13 Documentation ........................................................................................................................... 13 Moving to Live and Monitoring ................................................................................................... 14 Content Development ................................................................................................................. 14 Future Parser Modifications ....................................................................................................... 14 Managing the Device Product Lifecycle ..................................................................................... 14

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

About Us
EdgeSeven are Information Security specialists delivering world-class situational awareness capabilities to organizations that care about their data. We are NOT a broad security integrator or reseller. We are SIEM experts. We achieve results by utilizing the most sophisticated technology on earth, applying it intelligently and putting it in the hands of absolutely the best people. Having deployed some of the most complex and demanding implementations in the world we understand the importance of success, and our business is built around customer satisfaction and ensuring solutions meet your business goals both now and over the longer term.

Learning More About Flex Connectors

EdgeSeven has specifically built a comprehensive Flex Connector development training course. The course goes into great detail on understanding what Flex Connectors are and how to build them. The course is predominantly hands-on, which we believe allows for the best transfer of knowledge to the students. For more information and pricing please contact sales@edgeseven.com

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Introduction
ArcSight has a wide range of standard connectors that are available off the shelf that are able to correctly interpret the various manufacturers events and display the information in the respective event fields within ArcSight. Sometimes however the client may have a bespoke application or network device, for which ArcSight does not have a standard connector. In this instance we are able to leverage the Flex Connector Framework to build a custom connector that will allow these events to be captured. The flex connector is a substantial framework and has many methods available to collect event logs. This document provides a methodology to help choose, build and configure the appropriate flex connectors.

Building the Flex Connector

Familiarity with Flex Connectors
Before beginning the development of any flex connector it is recommended that you read and understand the flex connector configuration guide. This guide is available as part of the standard connector configuration guides pack available from the ArcSight support website. At the time of writing the latest version of the configuration guide is FlexConnectorDevConfig.pdf 01-Jul-2008

Information Gathering
The first step in building a flex connector is trying to gain an understanding of the device and the events it produces. To be able to produce an accurate flex connector you will need as many sample events as possible (preferably from a production environment). By having as many events as possible, you will be able to build a more complete regex/sql structure, minimizing the need to make amendments to the parser in the future as well as ensuring that a higher percentage of the events are processed correctly. It is also essential to have the product event log information guide. This will help when tokenizing and mapping the event severity. Without the information, it may not be clear what the information is actually referring to. For example id:23232 may mean userid 23232 or eventid 23232 or even session id 23232. Below shows an example of an event breakdown for a cisco WLC device:

%FACILITY-SEVERITY-MNEMONIC: Message-text

FACILITY code The facility code consists of two or more uppercase letters that indicate the facility to which the message refers. The facility code is preceded by a % sign. A facility can be a device, a protocol, or a module of the system software. Examples of facility codes are %BCAST (messages that are related to broadcasting) and %LWAPP (messages that are related to the Lightweight Access Point Protocol).
Security Situational Awareness Copyright Secmon Ltd 2012 WWW.EDGESEVEN.COM

SEVERITY level The severity level is a single-digit code from 0 to 7 that reflects the severity of the condition. The lower the number, the more serious the situation. The table below lists the message severity levels.

Severity Level 0 emergency 1 alert 2 critical 3 error 4 warning

Description System is unusable Immediate action required Critical condition Error condition Warning condition

MNEMONIC code The MNEMONIC code uniquely identifies the message. Message text Message text is a text string that describes the condition. The text string sometimes contains detailed information about the event, including terminal port numbers, network addresses, or addresses that correspond to locations in the system memory address space. Because variable fields change from message to message, they are represented here by short strings enclosed in square brackets ([ ]). A decimal number, for example, is represented as [dec]. Table 1-2 lists the variable fields in messages.

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Choosing the Flex Connector Type

ArcSight provides different types of flex connectors (see the table below) that allows collection from many different event sources as well as varying event formats (like delimited and free form). Choosing the right flex connector type is an essential part of building the required flex connector.
Flex Connector Type Operation

Flex Connector File

Use this for a single file if the log file is delimited and new events are appended to the end of the same file

Flex Connector Regex File

Use this for a single file if the log file is of a free form and events are appended to the end of the same file Use this for multiple files from the same device type if the log file is of a free format but the events are not appended to the end of the same file, but rather completed files are dropped in the folder location.

Flex Connector Regex Folder File For example files are that FTPed from another location into a directory, but the files are all from the same device and in the same format. Use this for multiple files of delimited or free format that either have events appended or files dropped. Each entry in the configuration table can be used for processing a different folder and format type. Flex Connector Multiple Folder File For example if you have a delimited file in folder a and a free format file in folder b. Instead of installing two connectors it can be handled by one using a separate configuration entry for each Flex Connector XML File Use this for multiple files if the event feed is in XML and the files are dropped into a folder

Flex Connector ID-Based DB

Use this if you have a single database that uses an increasing ID field to distinguish events

Flex Connector Time-Based DB

Use this if you have a single database that uses an increasing timestamp to distinguish events Use this if you have multiple instances for either time or id based events (please note that this connector can only support one Driver).

Flex Connector Multiple DB For example if you have you two Oracle databases, one that is ID based and one that is time based. Flex Connector SNMP Use this connector if you are receiving events via SNMP traps for a single OID. For each new OID, you will need to install a new connector

Syslog Daemon / File / Pipe

Use this connector if the events are sent to the connector using the SYSLOG protocol.

Note 1: There are other beta flex connectors that are not listed here Note 2: Please see page 25 of the flex connector config guide for names and locations of the various parser properties files.

As an example assume that we have three devices that are able to FTP logs. Two of the devices are the same OS version and model with events in free format. The third device is different from the first two but also sends events via FTP. Its log format is in a delimited format. Based on the information above we would select the Flex Connector Multiple Folder File and setup two entries in the configuration. The first entry would look in the folder where the first two devices are

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

sending logs and use the sdkrfilereader format. The second entry would look in the location where the third device is sending its logs and use the sdkfilereader format.

Basic Building Blocks

Now that we have a good understanding of the event information and have decided on what connector we are going to use, we can start an initial attempt at trying to conceptually parse the events. On most occasions the event format is simplistic enough to be able to determine which method will be used to parse the events (e.g. when using delimited files), however sometimes you will need to test various methods to determine which would be the most suited to extract the information that you would like to use. As an example, imagine that we have a database that we would like to pull information from. Every field in the database is unique and only holds a single bit of information except for the event_message field (see below). This field contains a long string about the event, but also contains additional information that we would like to capture and tokenise. In this instance we may have to use a Time or ID based connector and call an extra processor (sdkrfilereader type) to process the event_message field separately where we can obtain the Source IP Address
Event_ID 1 2 3 4 Event_User Test1 Test2 Test3 Test4 Event_Target 10.10.10.1 10.10.10.5 10.10.10.6 10.10.10.99 Event_Message Connection from 10.10.10.2 to 10.10.10.1 Opened connection from 10.10.10.9 Close connection from 10.10.10.8 Ssh connection from 10.10.10.45

Its good practice to run through a few events with other team members to ensure that a common name, labelling, severity, deviceEventClassId and categorization format are agreed upon.

Method of Attack
Once you have reviewed the possibilities for parsing the event information you will need to make a decision on which method you will use and then start the development of the actual parser. Of course this is not set in stone, and if you feel that after configuring the parser for some events it doesnt look like it will work well, then review and try an alternate method.

Creating the Parser

ArcSight provides two tools that are included in every connector installation to help with the flex connector development. The first is the flexagentwizard that is used to create parser properties files for delimited file types. The second is the regex wizard, which is used to create parser properties files for free form file types.

Using the FlexAgentWizard

To launch the flexagentwizard go to the bin directory of the respective connector and type arcsight flexagentwizard
root@server: cd <connector home>/current/bin root@server: ./arcsight flexagentwizard Security Situational Awareness Copyright Secmon Ltd 2012 WWW.EDGESEVEN.COM

You will see a screen that looks similar to below. You can review pages 67 onwards in the latest flex connector configuration guide for how to use this tool to develop a sdkfilereader type properties file.

Note: There are a few limitations that the interface has (but these can be changed in the properties files afterwards): You are not able to provide your own device vendor You are not able to provide token names You are not able to set deviceSeverity You are not able to set a deviceEventClassId format string You are not able to add any comments

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Using the Regex Wizard

To launch the regex wizard go to the bin directory of the respective connector and type arcsight regex
root@server: cd <connector home>/current/bin root@server: ./arcsight regex

You will see a screen that looks similar to below. You can review pages 38 onwards in the latest flex connector configuration guide for how to use this tool to develop a sdkrfilereader type properties file.

Creating Sub Messages

For some log formats it may be best to make use of sub messages within the parser. This allows the creation of a main regex parser to handle the static event content, and various sub message parsers to handle variable content. Consider the example events below:
2.2.2.2 FW 2002/10/01 18:12:12 2.2.2.2:80 (tcp) action=drop Message: User test1 opened connection to server1234 Rule 25 2.2.2.2 FW 2002/10/01 18:12:12 2.2.2.2:80 (tcp) action=drop Message: Encryption failed, username jsmith Rule5 2.2.2.2 VPN 2002/10/01 18:12:12 2.2.2.2:80 (tcp) action=drop Message: Tunnel built to 10.10.10.1

Instead trying to match the events with a single regex line, we can break the message up in smaller sub messages, and extract the information from those events separately in the parser.
Security Situational Awareness Copyright Secmon Ltd 2012 WWW.EDGESEVEN.COM

To be able to use the submessage function, we need to declare a submessage ID. The submessage ID is used to tell the parser what submessage section it needs to use to parse the event. A good guide to selecting a submessage ID is to look for something that would group messages together. For the above example events, a good submessage ID would be to use FW and VPN. The actual event that the submessages would handle would be everything after Message:. The rest of the event is static and can therefore be handle by the main regex parser.
# FlexAgent Regex Configuration File do.unparsed.events=true regex=(\\d+\\.\\d+\\.\\d+\\.\\d+)\\s(\\S+)\\s(\\d+/\\d+/\\d+\\s\\d+\:\\d+\:\\d+)\\s(\\d+\\.\\d+\\.\\d+\\.\\d +)\:(\\d+)\\s\\((\\S+)\\)\\saction\=(\\S+)\\sMessage\:\\s(.*) token.count=8 token[0].name=test_deviceaddress token[0].type=IPAddress token[1].name=test_module token[1].type=String token[2].name=test_date token[2].type=TimeStamp token[2].format=yyyy/MM/dd HH\:mm\:ss token[3].name=test_sourceip token[3].type=IPAddress token[4].name=test_sourceport token[4].type=Integer token[5].name=test_proto token[5].type=String token[6].name=test_action token[6].type=String token[7].name=test_message token[7].type=String submessage.messageid.token=test_module submessage.token=test_message event.name=__stringConstant("FW Event") event.sourcePort=test_sourceport event.deviceAction=test_action event.deviceReceiptTime=test_date event.sourceAddress=test_sourceip event.message=test_message event.deviceAddress=test_deviceaddress event.deviceProcessName=test_module event.transportProtocol=test_proto submessage.count=2 submessage[0].messageid=FW submessage[0].pattern.count=2 submessage[0].pattern[0].regex=User\\s(\\S+)\\sopened\\sconnection\\sto\\s(\\S+)\\sRule\\s(\\d+) submessage[0].pattern[0].fields=event.sourceUserName,event.destinationHostName,event.deviceCustomString1 submessage[0].pattern[0].extramappings=event.deviceCustomString1Label\=__stringConstant("Rule Number") submessage[0].pattern[1].regex=Encryption\\sfailed,\\susername\\s(\\S+)\\sRule(\\d+) submessage[0].pattern[1].fields=event.sourceUserName,event.deviceCustomString1 submessage[0].pattern[1].extramappings=event.deviceCustomString1Label\=__stringConstant("Rule Number") submessage[1].messageid=VPN submessage[1].pattern.count=1 submessage[1].pattern[0].regex=Tunnel built to (\\d+\\.\\d+\\.\\d+\\.\\d+) submessage[1].pattern[0].fields=event.destinationAddress submessage[1].pattern[0].types=IPAddress

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Creating deviceEventClassId
The deviceEventClassId is a method that ArcSight uses to create a unique identifier for each event. For example all the ArcSight internal agent messages are in the format agent:xxx where xxx represent a number. When tracking events using rules we are able to use these numbers as they are unique for each event. Although not mandatory, it is best practice to assign a unique deviceEventClassId to every message that is parsed by the parser. An added advantage of having this information is that we can use this field for the getter field when assigning the categorization. For the naming format generally it is recommended to use the same naming format that Arcsight uses i.e. <device object>:aaa:bbb, where aaa represents the submessage ID number and bbb the pattern number. Sometimes this is not possible as one pattern may pass 2 or more messages with different outcomes (see the examples below). In this instance you could append the outcome to the end of the deviceEventClassId.
Message = User x login failed Message = User x login succeeded deviceEventClassId = fwbox:01:01:Success deviceEventClassId = fwbox:01:01:Failure

Assigning Severity
Severity is an important part of the Threat Level Formula as well as for usage in reports that make use of device / event Severity. Although it is not specifically required in a flex connector parser it is best practice to assign a severity to each event. For the most part devices will assign a severity to each of the event messages it sends in some format, whether it be a literal string or an integer that represents a severity level. These can be mapped to deviceSeverity directly. Below shows some examples of a device setting severity:
2.2.2.2 10/10/2008 warn Message: This is a warning message 2.2.2.2 10/10/2008 crit Message: This is a critical message 2.2.2.2 10/10/2008 0 Message: This is a warning message 2.2.2.2 10/10/2008 1 Message: This is a critical message

For the above messages the device severity could be mapped similar to below (note that you will need to assign the respective token that maps these values to the event.deviceSeverity field before you can map the severity)
severity.map.veryhigh.if.deviceSeverity=crit,1 severity.map.medium.if.deviceSeverity=warn,0

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Creating Map Files

Map files are a way that we can set ArcSight event fields based on the information in another field. Essentially the Map file is a CSV file that functions much the same way as a categorization file in that it uses getters and setters. The map files are always located under <agent home>/current/user/agent/map. There are already some example map files in the folder that you can view to understand the format. A very simple example is if you do not use DNS for hostname to IP resolution. This can be handled in the map file. The structure of the map file would look something similar to below:
event.deviceCustomNumber1,set.event.deviceAddress,set.event.deviceHostName 001,12.12.12.12,dev1 002,20.20.20.20,dev2

Assigning Categorization
ArcSight makes use of categorization to group different events but that have a similar meaning. Take for instance the example events below. The events are different in format but still mean the same thing. Rather than having to match each event individually, we can group these events into categories and then search for those category groups.
2.2.2.1 server1 Message: User logged in successfully from 10.10.10.1 10 Feb 2000 server1: Mary logged on successfully Feb 10 10:10:15 authentication succeeded for user peter

For the above events we could set the Behaviour category to /Authentication/Verify and the Outcome category as /Success. Like wise if there were authentication failures we would just change the Outcome category to /Failure Like the parser and map files, the categorization file must also be created in a folder specific location for that connector. The correct directory location for the related parser categorization file is found under /current/user/agent/acp/categorizer/current/<vendor name>/<product name>.csv. Note that the directory and file name must be in lower case, and all spaces should be replaced by _ Please read the flex connector configuration guide on page 71 onwards for further examples.

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Testing
Testing is an integral part of deploying a flex connector into the production environment and must be completed before deploying the flex connector. Failure to test could result in messages not being parsed correctly as well as the possibility of affecting other parsers. Below shows some of the key areas that testing should focus on:

Ensuring that the parser does not affect other parsers (this is only specific to syslog subagents). If the subagent parser that you have created is similar to that of a standard subagent parser, your parser may match events from another device, or visa versa. Ensuring that the whole event is parsed and the correct data is place inside the expected tokens. This can be done using active channels and field sets within the Arcsight console. Ensuring that all deviceCustom and Flex fields have appropriate labels. This can be checked by using custom columns in an active channel, or by using the event inspector Ensuring that spelling is correct for all string constants. This can be done by using active channels Ensuring that event severity has been configured and is being applied to each of the events. This can be checked with an active channel Ensuring that each of the events has a deviceEventClassId set. This can be checked with an active channel Ensuring that each of the events has categorization set. This can be checked using with an active channel If you make use of aggregation ensure that the existing aggregation will not aggregate the new events incorrectly.

Documentation
As part of the flex connector delivery it is recommended that a member of the team create documentation pertaining to the parser. At a minimum the document should cover the following: Pre-installation checks (e.g. ensuring that ESM is up and the necessary change requests have been submitted) How to install the required connector (for the platform that you support) Where to copy the properties, categorization and map files How to start the connector General information about the parser Any configuration changes that need to be made in agent.properties Any configuration changes that need to be made to the connector properties (such as aggregation and turbo mode) Show token to event field mappings

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM

Moving to Live and Monitoring

Once all testing has been completed, and the test results have been reviewed the connector should be moved into production (assuming that all the necessary change requests have been completed etc). When moving into production it is always useful to get a feel for the events that are being sent to the connector beforehand. This way you can compare the results post parser installation to ensure that the new parser has had no affect on the current event feeds (only applicable to syslog subagents)

Content Development
Once the new parser has been moved into production and the events are being received, you can begin preparing the necessary content to be able to deliver the expected use cases.

Future Parser Modifications

If possible it is advisable to enable unparsed events and raw events to be collected by the connector (this is configured on the default page of the connector properties). This is useful for tracking any new messages that may not have been present in the original event log samples. If you have unparsed events and raw events enabled on the connector you can run a report on the manager to show all the unparsed events for that particular connector. Based on the report you would have the necessary information at hand to make any parser modifications to handle the unparsed events. Should it not be possible to enable unparsed events tracking it is still possible to track the unparsed events in the agent log files. To find unparsed events, open the agent.log file and search for parseValue. This message will be followed by the message that the connector was not able to parse correctly. Be aware that if you have low retention time (i.e. your logs are rotated quickly), you may not see all the unparsed messages as the log files would be overwritten.

Managing the Device Product Lifecycle

To enable the effective delivery for the team responsible for security event monitoring it is essential that they are made aware of any changes that may happen on the target device. Some changes may include patching, service pack application or even major configuration changes. All of these changes have a potential impact on the format of the event logs and even possibly the structure of the event, which would affect the security event monitoring teams ability to parse the events correctly. Should there be any changes to the device, the security event monitoring team should be given the opportunity to test the impact the changes would have on the event logs being sent to ArcSight, and where necessary update the parser with the relevant changes. In a worse case scenario if a patch (that affects the log format) is applied without the security team being involved, all events may not be parsed correctly. This would mean that none of the ArcSight content (such as rules to track intruders/ management reports etc) would work.

Security Situational Awareness

Copyright Secmon Ltd 2012

WWW.EDGESEVEN.COM