IBM Security QRadar April 28, 2016

Open Mic #13: Log Source Protocols

Panelists
Colin Hay – QRadar Ecosystem Team Lead
Chris Collins – Integration Team Lead – L3/Maintenance
Randika Upathilake – Integration Team Lead/Roadmap
Adam Frank – Principal Solutions Architect
Jeff Rusk – Development Manager, QRadar L3 Engineering
Jamie Wheaton – QRadar Integration WinCollect Team Lead
Joey Maher – Support Technical Lead
Jonathan Pechta – Support Technical Writer

Reminder: You must dial-in to the phone conference to listen

to the panelists. The web cast does not include audio.

• USA toll-free: 1-866-803-2145

• Participant passcode: 3744658
• Slides and additional dial in numbers: http://ibm.biz/Bd4eSW

NOTICE: By participating in this call, you give your irrevocable consent to IBM to
record any statements that you may make during the call, as well as to IBM's use
of such recording in any and all media, including for video postings on YouTube. If
you object, please do not connect to this call.
© 2016 IBM Corporation
Introduction
During this presentation, we discuss some of the basics on
how QRadar uses log source protocols to collect event data.
Each protocol in QRadar is unique, whether it be
configuration properties, error messages, and use cases for
data collection.

Participants: If you want us to stop to ask a question or discuss a topic

further, make sure you ask in the webcast chat.

© 2016 IBM Corporation 2

How many log source protocols are in QRadar?
There are currently ~52 log source protocols in QRadar that
are available to administrators. Some of these are used for
general purpose collection and some are product / vendor
specific. As of 7.2.6, we support 315 log source types out of
the box, with 174 of them supporting auto detection via traffic
analysis. There are basically three categories of protocols
and these will be the focus of today’s discussions.
Listening Protocols Polling Protocols Specialty Protocols (other)
• Syslog • JDBC • REST API Protocols
• TLS Syslog • Log File • Amazon AWS
• TCP Multiline Syslog • SMB Tail • Microsoft Office 365
• UDP Multiline Syslog • Subscription Protocols
• Syslog Redirect (SDEE)
© 2016 IBM Corporation 3
Protocols available in QRadar 7.2.6
• Ahnlab Policy Center JDBC • MQ JMS • VMWare vCloud Director
• Amazon AWS REST API • Netskope Active REST API • WinCollect
• Ariel REST API • ObserveIT JDBC • WinCollect File Forwarder
• Blue Coat WSS REST API • Office 365 REST API • WinCollect Juniper SBR
• Cisco NSEL • Okta REST API • WinCollect Microsoft DHCP
• EMC VMWare • Oracle Database Listener • WinCollect Microsoft ISA/NPS
• HTTP Receiver • Salesforce REST API • WinCollect Microsoft ISA/Forefront
• IBM Fiberlink REST API • SDEE TMG
• IBM Security Identity • Seculert Protection REST API • WinCollect Microsoft SQL
Manager JDBC • JDBC - SiteProtector • WinCollect NetApp Data ONTAP
• IBM SmartCloud Orchestrator • SMB Tail • Microsoft DHCP
REST API • SNMPv1 • Microsoft Security Event Log
• IBM Tivoli Endpoint • SNMPv2 • Microsoft Security Event Log
Manager SOAP Custom
• SNMPv3
• JDBC • Microsoft Security Event Log over
• Sourcefire Defense
• Sophos Enterprise MSRPC
Center Estreamer
Console JDBC • Microsoft Exchange
• Syslog
• Juniper Security Binary Log • Microsoft IIS
• Syslog Redirect
Collector • Forwarded
• TCP Multiline Syslog
• Juniper NSM • *Protocol Common
• TLS Syslog
• OPSEC/LEA
• UDP Multiline Syslog
• Log File
© 2016 IBM Corporation 4
 Events FAQ and terminology

© 2015 IBM Corporation

Events FAQ
What is an Event?
In QRadar terms, an event is a message we receive and process from a
device on your network, that represents the log of some particular action on
that device. An example would be an "ssh login" on a Linux server, a VPN
connection to a VPN device, or a firewall deny logged by your perimeter
firewall. These actions represent something that occurs at an instance of
time and are logged.

What is a log source protocol?

A log source protocol is the framework for getting event data off the wire.
This can include listening for data or complying with specifications to
establish connections, managing input queues, and getting the data to the
event pipeline (ECS).

© 2016 IBM Corporation 6

Events FAQ (continued)
What is a DSM?
A DSM is a Device Support Module. DSMs are created for security products,
software, or appliances to allow QRadar to understand (parse) the messages (event
format) provided by the device and categorize the data properly in the user interface.

What is Traffic Analysis (TA)?

Traffic Analysis, also known as Auto Discovery or Auto Detection, allows QRadar to
detect and create new log sources based on incoming data streams. Traffic Analysis
works with the Syslog protocol (both TCP and UDP) as well as SNMP protocol
messages. When QRadar starts receiving data from an new address or even from an
address we have already create data for, but another version of data, it sends that
data over to the "traffic analysis engine" for processing. If the data is not something
we recognize, an unsupported device, or if the data is sent at a very low rate (4-5
events per minute), we will likely fail auto discovery. Events from that device/system
will be assigned to the “SIM Generic-2” Log Source“ until the administrator resolves
the issue or creates a log source manually.

© 2016 IBM Corporation 7

How is the source IP or destination IP address determined?
When QRadar receives and processes event data, it must assign an IP address to the Source IP and Destination IP fields.
QRadar looks in the following locations, to determine the IP address to use, in the following order:
1. IP address fields in Payload Information
The availability of more detailed IP address information depends on each Log Source Type, as well as the events themselves,
as not all events will contain IP address fields. If the source IP address in available, the Source IP field will be updated with
this information. If source IP information is not available, then it will remain as it was last set in the previous step. The same is
true of destination IP information. If destination information is not available then it will remain set as either the Syslog
hostname field, if an IP was available, else it will remain set as the source of the packet.
2. The hostname field in the Syslog header
QRadar will look for an IP address in the hostname field of the Syslog header, if available. Note: Not all Syslog sources use
proper headers.
If an IP address is found, the Source IP and Destination IP fields are updated with this IP address. If the hostname field
contains a textual hostname, then it is not used. QRadar will not do a DNS lookup on a hostname, as it would take too much
time to do for every event, and would affect pipeline throughput capacity.
3. The source IP address of the packet the event came from, when received by QRadar
The Source IP and Destination IP fields are set to the source IP address of the packet itself. This would be the device that
sent the data to QRadar. If you are using an existing, centralized Syslog server to forward events to QRadar, you may often
see the IP address of the Syslog server in the Source IP and Destination IP fields. The best ways to avoid this is to do one of
the following:
• Set the Log Source device to send Syslog directly to QRadar.
• Preserve the initial Syslog headers, and have the originating devices configured to send an IP address in the
hostname filed of the Syslog header.
• Reconfigure your Syslog server to prepend a new Syslog header to the events it forwards to QRadar, with the
originating devices IP address in the hostname header field.

© 2016 IBM Corporation 8

Traffic Analysis in action!
Detecting new log source and adding it to the traffic analysis engine.
[ecs-ec] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=csd.ibm.lab:ecs-
ec/EC/TrafficAnalysis1/TrafficAnalysis]] com.q1labs.semsources.filters.trafficanalysis.TrafficAnalysisFilter:
[INFO] [NOT:0000006000][172.16.77.116/- -] [-/- -]Now collecting traffic analysis statistics for address
paloalto.paseries.test.com
Attempting to create new log source.
[ecs-ec] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=csd.ibm.lab:ecs-
ec/EC/TrafficAnalysis1/TrafficAnalysis]] com.q1labs.semsources.filters.trafficanalysis.TrafficAnalysisFilter:
[INFO] [NOT:0000006000][172.16.77.116/- -] [-/- -]Attempting to create a new sensor device: Log Source Type
= Contivityv2, Address = paloalto.paseries.test.com...
Pausing TA on that log source identifier (syslog host header)
[ecs-ec] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=csd.ibmlab:ecs-
ec/EC/TrafficAnalysis1/TrafficAnalysis]] com.q1labs.semsources.filters.trafficanalysis.TrafficAnalysisFilter:
[INFO] [NOT:0000006000][172.16.77.116/- -] [-/- -]Pausing traffic analysis on address
paloalto.paseries.test.com while waiting for another device to be created...

If a log source fails to auto discover / auto detect

[ecs-ec] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=csd.ibm.lab:ecs-
ec/EC/TrafficAnalysis1/TrafficAnalysis]] com.q1labs.semsources.filters.trafficanalysis.TrafficAnalysisFilter:
[WARN] [NOT:0070014101][172.16.77.116/- -] [-/- -]Unable to determine associated log source for IP address
<172.16.158.160>. Unable to automatically detect the associated log source for IP address.
© 2016 IBM Corporation 9
 Listening Protocols (Syslog)

© 2015 IBM Corporation

Standard fields in the log source user interface
Log sources that do not
auto discover in QRadar
must be manually created
by the administrator.

These are the common

fields for a Syslog
protocol log source.
Remember that a single
product might support
multiple protocols.

A common example is Syslog and TLS Syslog. Even though QRadar might
list two protocol options, it is up to the administrator to ensure that the
product is actually capable of providing data using the selected protocol.
© 2016 IBM Corporation 11
Standard fields in the log source user interface (continued)
• Log Source Name
• Log Source Description
• Log Source Type
• Protocol Configuration**
• Log Source Identifier**
• Enabled
• Credibility
• Target Event Collector**
• Coalescing Events**
• Incoming Payload Encoding
• Store Event Payload

© 2016 IBM Corporation 12

The Syslog Protocol
• QRadar’s Syslog port is TCP or UDP 514. For QRadar appliances that
collect or process events, TCP/UDP port 514 (ecs-ec) listens on all
interfaces. Ports are enabled when appliances are added to the
deployment.
• QRadar supports Syslog RFC3164 and RFC5424.
• Termination characters in a Syslog payload will truncate the event. QRadar
will then create a secondary event under SIM-Generic as “Stored” from the
remaining event payload data.
• UDP Syslog truncates payloads at 1024 bytes by default.
• TCP Syslog truncates payloads at 4096 bytes by default. This value is
configurable in the QRadar 7.2.6 System Settings.

© 2016 IBM Corporation 13

TLS Syslog Protocol
TLS Syslog is a secure protocol that encrypts
payload data for transport. TLS requires
certificates, which can either be provided to
QRadar or a default “generated” certificate can
be used. Private keys for provided certificate
path must be a DER-encoded PKCS8 key. The
configuration fails with any other key format.
One TLS log source can support up to 1,000
auto discovered TLS log sources on a single
appliance.

6514 is the default port per specification, however,

alternate ports are supported. A Deploy Full
Configuration is required if an saved/existing port is
changed.
To verify if the TLS port is listening, type:
netstat -tulpn | grep 6514

14 © 2016 IBM Corporation 14

TCP Multiline Syslog
The TCP Multiline Syslog protocol uses regular expressions to define the start, end, or stand
and end of a TCP Syslog payload that spans multiple lines to create a single-line event for
QRadar.

If you only specify an “Event Start Pattern” as your regular expression, the event will be
created using the first occurrence of the start pattern and then the next match that occurs from
the regex pattern you entered. Everything in
between is created as a single-line event.
This same logic applies to only having an
Event End Pattern defined in the log
source configuration.

Depending on your event payload, you might

not require both a start and end event pattern.

The Event Formatter is only used for

specific scenarios, such as TCP Syslog event
payloads from a system sending Windows
Multiline data.

© 2016 IBM Corporation 15

UDP Multiline Syslog
The UDP Multiline Syslog protocol uses regular expressions to define a repeating value
from the UDP Syslog payload to logically create a single-line event for QRadar.

<134>May 14 19:04:49 DB1 [25646]: 2065699 LOG: data_request user: testadmin DB: PCI-2
<134>May 14 19:04:49 DB1 [25646]: 2065699 select
o.id,o.type,o.serial_number,o.card_number,o.currency_id,to_char,o.sum_in

NOTE: The flat file version of this same

functionality can be found in the Log File
Protocol using the
Event Generator: ID-Linked Multiline.

The Event Formatter is only used for

specific appliances, such as Cisco ACS
Multiline event data.

© 2016 IBM Corporation 16

Syslog Redirect
The purpose of the Syslog Redirect is to deal with log sources that either do not correctly set a
syslog header, providing the wrong log source identifier in the header, or a log source identifier
that messes up our internal message routing (ie. 127.0.0.1).

Syslog Redirect takes the value from the Log Source Identifier RegEx and the matched value
(if anything) to substitute that value in as the log source identifier internally (as the packet IP).
If the event has a valid syslog header we are overriding it with a new one that contains the
new origin address. If an event does not have a match to the RegEx it should be posted as a
regular syslog event and passed to ECS in its unmodified state.

Syslog Redirect matched event = <Redirect header with injected value><default Syslog header><Payload for event data>

Good for correcting systems that

obscure the true event source IP:
- Load balancers
- SymantecServer
- Syslog forwarders / Syslog relay
- Virtual systems
- Log aggregators / 3rd party hosts
© 2016 IBM Corporation 17
 Polling Protocols (JDBC / Log File)

© 2015 IBM Corporation

JDBC Protocol
The JDBC protocol can retrieve data from
MSDE, Postgres, Oracle, Sybase, DB2,
or Informix databases. The protocol can
query true databases or views created by
administrators specifically for QRadar
polling.

Requires an incrementing “Compare

Field” value to work properly. This can be
an ID, timestamp, or any increasing
numeric value that is not reset.

The log source identifier must be unique.

You cannot have two log sources with
DBname1@mydatabase. However, you
can specify a table to make the value
unique:
table|databasename@HostnameorIP

© 2016 IBM Corporation 19

JDBC Protocol (continued)
Predefined queries can be used to collect
data from specific databases. Instead of
requiring the admin to create a view, the
predefined query can send complex
commands to retrieve data easily.

Select list is the data being retrieved

during the poll of the database.

EPS throttle is set by default to 20,000

Events per second. This value can
exceed your license, but very large initial
polls can cause performance issues.

If your database requires or forces an

encryption check to connect, the Use SSL
check box is required.

© 2016 IBM Corporation 20

Log File Protocol
The Log File protocol is a file retrieval
system to use FTP, SFTP, SCP, or AWS to
connect to a remote log repository or
appliance and copy the files to /store/tmp in
QRadar for processing.
The Log File protocol is not intended for files
that append data, but instead for files that
rollover or are created new on an interval,
such as a timestamp.

Ignore Previously Processed File(s) is

used to track each file that has been
imported to ensure that the file is not
processed a second time, which would
create duplicate files. If you uncheck this
box, the Log File protocol will continue to
import and process all files that match the
File Pattern / regular expression.
© 2016 IBM Corporation 21
Log File Protocol (continued)
Event Generators allow QRadar to handle
special formatting scenarios that might be in
the log file or logs that need special handling
or processing before they can be sent to be
parsed in ECS.
If Run on Save is checked, the Log File
attempts to retrieve data immediately after
the log source is saved.

Tip: When you see the term “Pattern” in

QRadar this almost always indicates a
regular expression pattern is required and
not a glob, such as * (wildcard) value.

Recurrence defines the frequency with the

protocol attempts to retrieve data. The
shortest possible interval is 15M.

© 2016 IBM Corporation 22

Log File Protocol (continued)
Remote Directory defines the file location of the
logs. If the path for the remote directory is left blank,
the protocol starts from the user’s home directory.
You logs must reside in this directory or a sub-
directory to be successful if you leave the remote
directory blank.
Recursive allows the protocol to traverse sub-
folders of the remote directory to look for files that
match the file pattern (regex).
Processors are used to extract binary files, such as
zip, tar, tgz, etc before writing the file to /store/tmp
on the QRadar appliance.
Folder Separator is used when the remote file
system uses non-standard separators. The Log File
protocol can use the correct separator to navigate
the remote file system.

© 2016 IBM Corporation 23

SMB Tail Protocol
The SMB Tail protocol allows administrators to receive new data appended to a file on a
remote Samba share. The protocol keeps track of the file size or data modified time for any
files that match the File Pattern (regular expression).

The SMB Tail protocol is the base for a number of other QRadar protocols, such as
Windows IIS protocol, Microsoft DHCP protocol, Microsoft Exchange protocol, and more.

There are many use cases for SMB Tail. In general we utilize SMB Tail for Windows
applications that do not have their own method for sending their events, such as Syslog.
The Windows operating system also needs to have a 'share' configured to ensure that
Samba can connect and read the files. For example, we use SMB Tail for things like
Microsoft Exchange, Microsoft DHCP, Microsoft IIS, etc.

TIP: To run the smbclient tool, type /usr/bin/smbclient -L host

Where host is the server you are attempting to connect to. Alternately, you can use
/user/bin/smbclient \\\\netbios_name\\c$ -U domain\user

© 2016 IBM Corporation 24

 Tips and Performance Suggestions

© 2015 IBM Corporation

Tips and performance suggestions
1. When using the Log File protocol, query in large directory structures can take
longer than expected when you attempt to recursively search. The protocol will
attempt to traverse the entire directory structure to locate files. Having your event
log files in a dedicated directory structure is never a bad idea.

2. Disabling functional log sources for protocols like JDBC can unintentionally cause
license spikes and performance. If an administrator configures a polling log
source, like JDBC or Log File protocol, when re-enbabled these protocols can pull
in massive amounts of data. So, be wary of disabling log sources long term.

3. Creating a unique QRadar user is usually a good idea. This allows you to audit
the data that QRadar log source protocols are retrieving by providing a unique
user name.

4. Remember that if you install a protocol manually, the instructions in the command-
line inform administrators to: 1. Complete a Deploy Full Configuration, then
2. Restart Tomcat (Admin tab > Advanced > Web Server Restart).

© 2016 IBM Corporation 26

Tips and performance suggestions (continued)
5. When you have multiple log sources defined in parsing order, it will try the first one, and
if it doesn't match, tries the next. If it does not match the second or next value, it will
then store that event to the Primary log source as a stored event.. If it does match the
second, it will then store properly as the second.
6. Some event sources do not allow admins to configure Syslog ports below TCP 1024,
for example like Cisco ASA. If you have a scenario like this, you can use iptables in
QRadar to preroute and redirect data from a non-standard port to QRadar’s supported
Syslog port, which is TCP / UDP 514. However, you cannot change QRadar’s default
Syslog port.
7. Log sources can be bulk added, but only using select protocols such as Syslog and
Windows protocols (WinCollect, MSRPC, and WMI).
8. If you have older log sources, it can be better to disable them than to delete them. The
reason behind this is that in both circumstances the data is still on the QRadar
appliance. However, if the log source is deleted, you cannot quick filter by log source
group or log source name as those references no longer exist. If you think you might
want to look up older data, think about disabling older log sources. Also, deleted log
sources can auto discover again if they continue to send event data.

© 2016 IBM Corporation 27

 Specialty Protocols (APIs)

© 2015 IBM Corporation

Specialty Protocols – Amazon AWS CloudTrail
In the Amazon AWS log source
configuration, the Access Key ID is
equivalent to the Public Key field. The
Secret Access Key as defined in
Amazon's AWS documentation
should be configured to use the
Access Key field in the log source
configuration.

Copy the .DER certificate to the

opt/QRadar/conf/trusted_certificates
directory of the QRadar appliance
that manages the Amazon AWS
CloudTrail log source.

© 2016 IBM Corporation 29

Specialty Protocols – What do you want to talk about?

© 2016 IBM Corporation 30

 Questions & discussion

© 2015 IBM Corporation

Advanced questions from the forums
Q1: I know we can use the Microsoft Security Event Log over MSRPC to pull events that are
written to the default Windows Logs (e.g. Application, Security, Setup, System....). Will there
be any way to pull events that have been configured to write under the Applications and
Services Logs using a preloaded protocol? I don't know of any way besides configuring a
WinCollect agent to manually pull these logs. This would be a nice to have since there are
instances where we configure logs to be written to a separate location to keep them separate
from all the other "noise".

Q2: What is the best solution to retrieve logs from file located on a Windows shared folder?

Q3: When using Log File protocol to retrieve a log from a Unix file:
If I check "Ignore Previously Processed File(s)" I will loose then some logs from the latest open file
where new logs are written.
If "Ignore Previously Processed File(s)" is not checked, the log source will always retrieve the same
events and logs will be duplicated.

© 2016 IBM Corporation 32

Questions for the panel?
Now is your opportunity to ask questions of our panelists.

To ask a question now:

Press *1 to ask a question over the phone

or

Type your question into the SmartCloud Meetings chat

To ask a question after this presentation:

You can ask questions in our General forum:
https://www.ibm.com/developerworks/community/forums/html/topic?id=16602fca-
978a-403e-83ae-d4edbebab3ec&ps=25

© 2016 IBM Corporation 33

Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside
your enterprise. Improper access can result in information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems, including for use in attacks
on others. No IT system or product should be considered completely secure and no single product, service or security measure can be completely effective in preventing improper use or access.
IBM systems, products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other
systems, products or services to be most effective. IBM DOES NOT WARRANT THAT ANY SYSTEMS, PRODUCTS OR SERVICES ARE IMMUNE FROM, OR WILL MAKE YOUR ENTERPRISE
IMMUNE FROM, THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.

THANK YOU
www.ibm.com/security

© Copyright IBM Corporation 2015. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any
kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor
shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use
of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and / or
capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product
or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries
or both. Other company, product, or service names may be trademarks or service marks of others.