[syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

[Tim Ghetti]

Hi - having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don't see the messages being parsed. Running pdbtool against the logs seems to work.

# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
LEGACY_MSGHDR=%ASA-6-305011:
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system



**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };


**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
<pattern>%ASA</pattern>
<description>
This ruleset covers the Cisco ASA firewalls
</description>
<rules>
<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">
<!-- support: 1594 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>
<!-- support: 1369 -->
<patterns>
<pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message>
</example>
</examples>
</rule>
<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>
<!-- support: 1254 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message>
</example>
</examples>
</rule>
<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>
<!-- support: 1256 -->
<patterns>
<pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built outbound UDP connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>
<!-- support: 1594 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>
<!-- support: 1763 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound UDP connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>
<!-- support: 1539 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message>
</example>
</examples>
</rule>
<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>
<!-- support: 3648 -->
<patterns>
<pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule provider='%ASA' class='system' id='39'>
<patterns>
<pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>
<pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>

[Evan Rempel]

Welcome to the horrible world of Cisco logging :-(

The issue you are bumping into is that when these log lines are parsed
by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is
no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are
so many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact
log lines.

Cisco can add a * to the date/time stamp to indicate that there is no
time server configured on the device.
That makes the date/time invalid as far as parsing is concerned.

Cisco can add a period (.) to the date/time stamp to indicate that there
is a time server configured on
the device, but the time server can not be reached. Again, this makes
the date/time invalid as far as parsing
is concerned.

Cisco can add a sequence number at the start of the log line rather than
starting the line with a date/6time stamp.
Invalid parsing again.

Cisco can leave out the sequence number but still include the trailing
colon from the sequence number.

You get the idea. So many ways to get it wrong, and they never get it
right :-(

Evan.

Hi – having some trouble getting paterndb functional and looking for
some help. I would like to use patterndb to parse my cisco ass
firewall logs before sending it to elasticsearch. However when the
messages get to elasticsearch, I don’t see the messages being parsed.
Running  pdbtool against the logs seems to work.
# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f
/var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to
OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system
**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
    elasticsearch2(
        client-mode("http")
        cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
        cluster-url("http://X.X.X.X:9200")
        type("syslog")
        flush-limit("1")
    );
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa
{db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa);
destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };
**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
  <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
  <pattern>%ASA</pattern>
  <description>
    This ruleset covers the Cisco ASA firewalls
  </description>
    <rules>
      <rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86"
class="system">
        <!-- support: 1594 -->
        <patterns>
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
@IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown ICMP connection
for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system'
provider='%ASA'>
        <!-- support: 1369 -->
        <patterns>
@***@ANYSTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown dynamic UDP
translation from any:X.X.X.X/X to outside:X.X.X.X/X duration
0:00:00</test_message>
            </example>
        </examples>
      </rule>
      <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system'
provider='%ASA'>
        <!-- support: 1254 -->
        <patterns>
@for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown UDP connection
55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00
bytes 132</test_message>
            </example>
        </examples>
      </rule>
      <rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system'
provider='%ASA'>
        <!-- support: 1256 -->
        <patterns>
@(@ESTRING::)@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built outbound UDP
connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to
inside:X.X.X.X/X (X.X.X.X/X)</test_message>
            </example>
        </examples>
      </rule>
      <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system'
provider='%ASA'>
        <!-- support: 1594 -->
        <patterns>
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
@IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built inbound ICMP
connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr
X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system'
provider='%ASA'>
        <!-- support: 1763 -->
        <patterns>
@(@ESTRING::)@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built inbound UDP
connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to
identity:X.X.X.X/X (X.X.X.X/X)</test_message>
            </example>
        </examples>
      </rule>
      <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system'
provider='%ASA'>
        <!-- support: 1539 -->
        <patterns>
@for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown UDP connection
55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01
bytes 88</test_message>
            </example>
        </examples>
      </rule>
      <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system'
provider='%ASA'>
        <!-- support: 3648 -->
        <patterns>
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to
@ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built dynamic TCP
translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule provider='%ASA' class='system' id='39'>
        <patterns>
@src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type
@NUMBER::@, code @NUMBER::@</pattern>
        </patterns>
      </rule>
    </rules>
  </ruleset>
</patterndb>

[Tim Ghetti]

Ok, that makes sense. I sort of suspected that this was due to the program name not matching, since pdbtool works when specifying the program name. Do you know if there is a way to configure patterndb so that it matches the program name, regardless of the full program name with event code? Is it possible to regex the program or use the patterndb format i.e. <pattern>%***@ANYSTRING::@</pattern> or something similar?

Tim

From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Evan Rempel
Sent: Tuesday, February 27, 2018 8:26 PM
To: syslog-***@lists.balabit.hu
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

Welcome to the horrible world of Cisco logging :-(

The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are so many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log lines.

Cisco can add a * to the date/time stamp to indicate that there is no time server configured on the device.
That makes the date/time invalid as far as parsing is concerned.

Cisco can add a period (.) to the date/time stamp to indicate that there is a time server configured on
the device, but the time server can not be reached. Again, this makes the date/time invalid as far as parsing
is concerned.

Cisco can add a sequence number at the start of the log line rather than starting the line with a date/6time stamp.
Invalid parsing again.

Cisco can leave out the sequence number but still include the trailing colon from the sequence number.

You get the idea. So many ways to get it wrong, and they never get it right :-(

Evan.

On 02/27/2018 04:46 PM, Tim Ghetti wrote:
Hi – having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don’t see the messages being parsed. Running pdbtool against the logs seems to work.

# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
LEGACY_MSGHDR=%ASA-6-305011:
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system



**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };


**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
<pattern>%ASA</pattern>
<description>
This ruleset covers the Cisco ASA firewalls
</description>
<rules>
<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">
<!-- support: 1594 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>
<!-- support: 1369 -->
<patterns>
<pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message>
</example>
</examples>
</rule>
<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>
<!-- support: 1254 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message>
</example>
</examples>
</rule>
<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>
<!-- support: 1256 -->
<patterns>
<pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built outbound UDP connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>
<!-- support: 1594 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>
<!-- support: 1763 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound UDP connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>
<!-- support: 1539 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message>
</example>
</examples>
</rule>
<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>
<!-- support: 3648 -->
<patterns>
<pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule provider='%ASA' class='system' id='39'>
<patterns>
<pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>
<pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>

[Evan Rempel]

The program name will have an implied @ANYSTRING@ on the end, so if the
syslog payload could
be parsed to detect the program name of %ASA..... then your
patterndb.xml would work.
The problem is that they syslog payload can not be parsed.

What we do is use a hand crafted patterndb to detect all of the
different problem formats
that Cisco logs (in our environment) and change

a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is
the leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.

All of the poor hosts, sequence numbers, timestamps etc are all thrown away.

After that, the internal buffers of syslog-ng contain a usable PROGRAM
and MESSAGE such that we can use
a patterndb to match the message part of the log line.

This does mean that we have two patterndb parsers for every log line but
it seems to work well for us.
What would work better is if Cisco would fix their logging, but that
isn't going to happen in my lifetime :-(

I hope that makes sense.

Evan.

Post by Tim Ghetti
Ok, that makes sense. I sort of suspected that this was due to the
program name not matching, since pdbtool works when specifying the
program name. Do you know if there is a way to configure patterndb so
that it matches the program name, regardless of the full program name
with event code? Is it possible to regex the program or use the
something similar?
Tim
Behalf Of *Evan Rempel
*Sent:* Tuesday, February 27, 2018 8:26 PM
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
Welcome to the horrible world of Cisco logging :-(
The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there
is no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are
so many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log lines.
Cisco can add a * to the date/time stamp to indicate that there is no
time server configured on the device.
That makes the date/time invalid as far as parsing is concerned.
Cisco can add a period (.) to the date/time stamp to indicate that
there is a time server configured on
the device, but the time server can not be reached. Again, this makes
the date/time invalid as far as parsing
is concerned.
Cisco can add a sequence number at the start of the log line rather
than starting the line with a date/6time stamp.
Invalid parsing again.
Cisco can leave out the sequence number but still include the trailing
colon from the sequence number.
You get the idea. So many ways to get it wrong, and they never get it right :-(
Evan.
Hi – having some trouble getting paterndb functional and looking
for some help. I would like to use patterndb to parse my cisco ass
firewall logs before sending it to elasticsearch. However when the
messages get to elasticsearch, I don’t see the messages being
parsed. Running  pdbtool against the logs seems to work.
# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA
-f /var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system
**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
    elasticsearch2(
        client-mode("http")
        cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
        cluster-url("http://X.X.X.X:9200")
        type("syslog")
        flush-limit("1")
    );
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa
{db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa);
destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };
**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
  <ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
  <pattern>%ASA</pattern>
  <description>
    This ruleset covers the Cisco ASA firewalls
  </description>
    <rules>
      <rule provider="%ASA"
id="b3de7699-8213-c744-944e-9413298afe86" class="system">
        <!-- support: 1594 -->
        <patterns>
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
@IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown ICMP
connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr
X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6'
class='system' provider='%ASA'>
        <!-- support: 1369 -->
        <patterns>
@***@ANYSTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown dynamic UDP
translation from any:X.X.X.X/X to outside:X.X.X.X/X duration
0:00:00</test_message>
            </example>
        </examples>
      </rule>
      <rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87'
class='system' provider='%ASA'>
        <!-- support: 1254 -->
        <patterns>
@ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown UDP
connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X
duration 0:00:00 bytes 132</test_message>
            </example>
        </examples>
      </rule>
      <rule id='00c0732d-1e34-7340-a75f-21198bf71137'
class='system' provider='%ASA'>
        <!-- support: 1256 -->
        <patterns>
@(@ESTRING::)@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built outbound UDP
connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to
inside:X.X.X.X/X (X.X.X.X/X)</test_message>
            </example>
        </examples>
      </rule>
      <rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d'
class='system' provider='%ASA'>
        <!-- support: 1594 -->
        <patterns>
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr
@IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built inbound ICMP
connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr
X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule id='8be7928d-66e7-7042-abd5-869d6b49c56e'
class='system' provider='%ASA'>
        <!-- support: 1763 -->
        <patterns>
@(@ESTRING::)@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built inbound UDP
connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to
identity:X.X.X.X/X (X.X.X.X/X)</test_message>
            </example>
        </examples>
      </rule>
      <rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21'
class='system' provider='%ASA'>
        <!-- support: 1539 -->
        <patterns>
@ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to
@***@ANYSTRING::@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Teardown UDP
connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X
duration 0:02:01 bytes 88</test_message>
            </example>
        </examples>
      </rule>
      <rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b'
class='system' provider='%ASA'>
        <!-- support: 3648 -->
        <patterns>
@ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
        </patterns>
        <examples>
            <example>
                <test_message program='%ASA'>Built dynamic TCP
translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
            </example>
        </examples>
      </rule>
      <rule provider='%ASA' class='system' id='39'>
        <patterns>
          <pattern>regular translation creation failed for
        </patterns>
      </rule>
    </rules>
  </ruleset>
</patterndb>
Nￜnￜrￜￜￜￜ)emￜhￜyhiםￜw^ￜￜ

[Tim Ghetti]

Thanks!! It sounds like this is part of my problem. If you would be so kind as to send an example (or point me in the right direction) of how you are changing the program name and message body for the cisco logs that would be extremely helpful.

I also suspect that there may be another issue. I’m almost certain that my parser should be picking up at least some of the logs but I am not seeing any of the fields in ES showing up based on parsed logs. Assuming that patterndb is working, would the below config be correct for getting patterndb parsed variables to show as fields in ES?


destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};



From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Evan Rempel
Sent: Tuesday, February 27, 2018 9:06 PM
To: syslog-***@lists.balabit.hu
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

The program name will have an implied @ANYSTRING@ on the end, so if the syslog payload could
be parsed to detect the program name of %ASA..... then your patterndb.xml would work.
The problem is that they syslog payload can not be parsed.

What we do is use a hand crafted patterndb to detect all of the different problem formats
that Cisco logs (in our environment) and change

a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.

All of the poor hosts, sequence numbers, timestamps etc are all thrown away.

After that, the internal buffers of syslog-ng contain a usable PROGRAM and MESSAGE such that we can use
a patterndb to match the message part of the log line.

This does mean that we have two patterndb parsers for every log line but it seems to work well for us.
What would work better is if Cisco would fix their logging, but that isn't going to happen in my lifetime :-(

I hope that makes sense.

Evan.

On 02/27/2018 05:37 PM, Tim Ghetti wrote:
Ok, that makes sense. I sort of suspected that this was due to the program name not matching, since pdbtool works when specifying the program name. Do you know if there is a way to configure patterndb so that it matches the program name, regardless of the full program name with event code? Is it possible to regex the program or use the patterndb format i.e. <pattern>%***@ANYSTRING::@</pattern> or something similar?

Tim

From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Evan Rempel
Sent: Tuesday, February 27, 2018 8:26 PM
To: syslog-***@lists.balabit.hu<mailto:syslog-***@lists.balabit.hu>
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

Welcome to the horrible world of Cisco logging :-(

The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are so many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log lines.

Cisco can add a * to the date/time stamp to indicate that there is no time server configured on the device.
That makes the date/time invalid as far as parsing is concerned.

Cisco can add a period (.) to the date/time stamp to indicate that there is a time server configured on
the device, but the time server can not be reached. Again, this makes the date/time invalid as far as parsing
is concerned.

Cisco can add a sequence number at the start of the log line rather than starting the line with a date/6time stamp.
Invalid parsing again.

Cisco can leave out the sequence number but still include the trailing colon from the sequence number.

You get the idea. So many ways to get it wrong, and they never get it right :-(

Evan.

On 02/27/2018 04:46 PM, Tim Ghetti wrote:
Hi – having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don’t see the messages being parsed. Running pdbtool against the logs seems to work.

# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
LEGACY_MSGHDR=%ASA-6-305011:
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system



**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };


**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
<pattern>%ASA</pattern>
<description>
This ruleset covers the Cisco ASA firewalls
</description>
<rules>
<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">
<!-- support: 1594 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>
<!-- support: 1369 -->
<patterns>
<pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message>
</example>
</examples>
</rule>
<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>
<!-- support: 1254 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101037 for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message>
</example>
</examples>
</rule>
<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>
<!-- support: 1256 -->
<patterns>
<pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built outbound UDP connection 55101037 for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>
<!-- support: 1594 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>
<!-- support: 1763 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound UDP connection 55101078 for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>
<!-- support: 1539 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101084 for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message>
</example>
</examples>
</rule>
<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>
<!-- support: 3648 -->
<patterns>
<pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule provider='%ASA' class='system' id='39'>
<patterns>
<pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>
<pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>





Nￜnￜrￜￜￜￜ)emￜhￜyhiםￜw^ￜￜ

[Scheidler, Balázs]

One additional feature you might be interested in is our cisco-parser()
feature.

It basically expects a flags(no-parse) input and then puts things in the
right buckets, while recognising the various formats that cisco uses.

It would be appreciated if you could check if it works with your setup and
I can help fixing it up if it doesn't.

The parser got added to recent syslog-ng versions, so you might need to
upgrade to a new version.

Bazsi

Post by Tim Ghetti
Thanks!! It sounds like this is part of my problem. If you would be so
kind as to send an example (or point me in the right direction) of how you
are changing the program name and message body for the cisco logs that
would be extremely helpful.
I also suspect that there may be another issue. I’m almost certain that my
parser should be picking up at least some of the logs but I am not seeing
any of the fields in ES showing up based on parsed logs. Assuming that
patterndb is working, would the below config be correct for getting
patterndb parsed variables to show as fields in ES?
destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};
Of *Evan Rempel
*Sent:* Tuesday, February 27, 2018 9:06 PM
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
syslog payload could
be parsed to detect the program name of %ASA..... then your patterndb.xml would work.
The problem is that they syslog payload can not be parsed.
What we do is use a hand crafted patterndb to detect all of the different problem formats
that Cisco logs (in our environment) and change
a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the
leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.
All of the poor hosts, sequence numbers, timestamps etc are all thrown away.
After that, the internal buffers of syslog-ng contain a usable PROGRAM and
MESSAGE such that we can use
a patterndb to match the message part of the log line.
This does mean that we have two patterndb parsers for every log line but
it seems to work well for us.
What would work better is if Cisco would fix their logging, but that isn't
going to happen in my lifetime :-(
I hope that makes sense.
Evan.
Ok, that makes sense. I sort of suspected that this was due to the program
name not matching, since pdbtool works when specifying the program name. Do
you know if there is a way to configure patterndb so that it matches the
program name, regardless of the full program name with event code? Is it
possible to regex the program or use the patterndb format i.e.
Tim
*Sent:* Tuesday, February 27, 2018 8:26 PM
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch
Welcome to the horrible world of Cisco logging :-(
The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is
no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are so
many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log lines.
Cisco can add a * to the date/time stamp to indicate that there is no time
server configured on the device.
That makes the date/time invalid as far as parsing is concerned.
Cisco can add a period (.) to the date/time stamp to indicate that there
is a time server configured on
the device, but the time server can not be reached. Again, this makes the
date/time invalid as far as parsing
is concerned.
Cisco can add a sequence number at the start of the log line rather than
starting the line with a date/6time stamp.
Invalid parsing again.
Cisco can leave out the sequence number but still include the trailing
colon from the sequence number.
You get the idea. So many ways to get it wrong, and they never get it right :-(
Evan.
Hi – having some trouble getting paterndb functional and looking for some
help. I would like to use patterndb to parse my cisco ass firewall logs
before sending it to elasticsearch. However when the messages get to
elasticsearch, I don’t see the messages being parsed. Running pdbtool
against the logs seems to work.
# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f
/var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to
OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system
**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa {db-parser(file("/etc/syslog-
ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa);
destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };
**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
<pattern>%ASA</pattern>
<description>
This ruleset covers the Cisco ASA firewalls
</description>
<rules>
<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">
<!-- support: 1594 -->
<patterns>
@laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown ICMP connection for
faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>
<!-- support: 1369 -->
<patterns>
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown dynamic UDP
translation from any:X.X.X.X/X to outside:X.X.X.X/X duration
0:00:00</test_message>
</example>
</examples>
</rule>
<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>
<!-- support: 1254 -->
<patterns>
@ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@
@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection
55101037 <(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X
duration 0:00:00 bytes 132</test_message>
</example>
</examples>
</rule>
<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>
<!-- support: 1256 -->
<patterns>
@(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built outbound UDP connection
55101037 <(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to
inside:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>
<!-- support: 1594 -->
<patterns>
@laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound ICMP connection
for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>
<!-- support: 1763 -->
<patterns>
@for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound UDP connection
55101078 <(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to
identity:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>
<!-- support: 1539 -->
<patterns>
@***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection
55101084 <(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X
duration 0:02:01 bytes 88</test_message>
</example>
</examples>
</rule>
<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>
<!-- support: 3648 -->
<patterns>
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@
@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built dynamic TCP translation
from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule provider='%ASA' class='system' id='39'>
<patterns>
@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>
@ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@,
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>
Nￜnￜrￜￜￜￜ)emￜhￜyhiםￜw^ￜￜ
____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?
product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

[Tim Ghetti]

I tried using the cisco-parser and not having any luck with that either. When I enable the parser, I actually stop seeing outbound traffic to elasticsearch. My config is below


log {
source { udp(flags(no-parse)); };
parser { cisco-parser(); };
destination {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://192.168.101.199:9200 http://192.168.101.198:9200")
type("syslog")
flush-limit("1"));
};

}


[***@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Scheidler, Balázs
Sent: Tuesday, February 27, 2018 11:06 PM
To: Syslog-ng users' and developers' mailing list <syslog-***@lists.balabit.hu>
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

One additional feature you might be interested in is our cisco-parser() feature.

It basically expects a flags(no-parse) input and then puts things in the right buckets, while recognising the various formats that cisco uses.

It would be appreciated if you could check if it works with your setup and I can help fixing it up if it doesn't.

The parser got added to recent syslog-ng versions, so you might need to upgrade to a new version.

Bazsi

On Feb 27, 2018 22:29, "Tim Ghetti" <***@targetedsupport.com<mailto:***@targetedsupport.com>> wrote:
Thanks!! It sounds like this is part of my problem. If you would be so kind as to send an example (or point me in the right direction) of how you are changing the program name and message body for the cisco logs that would be extremely helpful.

I also suspect that there may be another issue. I’m almost certain that my parser should be picking up at least some of the logs but I am not seeing any of the fields in ES showing up based on parsed logs. Assuming that patterndb is working, would the below config be correct for getting patterndb parsed variables to show as fields in ES?


destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};



From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu<mailto:syslog-ng-***@lists.balabit.hu>] On Behalf Of Evan Rempel
Sent: Tuesday, February 27, 2018 9:06 PM
To: syslog-***@lists.balabit.hu<mailto:syslog-***@lists.balabit.hu>
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

The program name will have an implied @ANYSTRING@ on the end, so if the syslog payload could
be parsed to detect the program name of %ASA..... then your patterndb.xml would work.
The problem is that they syslog payload can not be parsed.

What we do is use a hand crafted patterndb to detect all of the different problem formats
that Cisco logs (in our environment) and change

a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.

All of the poor hosts, sequence numbers, timestamps etc are all thrown away.

After that, the internal buffers of syslog-ng contain a usable PROGRAM and MESSAGE such that we can use
a patterndb to match the message part of the log line.

This does mean that we have two patterndb parsers for every log line but it seems to work well for us.
What would work better is if Cisco would fix their logging, but that isn't going to happen in my lifetime :-(

I hope that makes sense.

Evan.

On 02/27/2018 05:37 PM, Tim Ghetti wrote:
Ok, that makes sense. I sort of suspected that this was due to the program name not matching, since pdbtool works when specifying the program name. Do you know if there is a way to configure patterndb so that it matches the program name, regardless of the full program name with event code? Is it possible to regex the program or use the patterndb format i.e. <pattern>%***@ANYSTRING::@</pattern> or something similar?

Tim

From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Evan Rempel
Sent: Tuesday, February 27, 2018 8:26 PM
To: syslog-***@lists.balabit.hu<mailto:syslog-***@lists.balabit.hu>
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

Welcome to the horrible world of Cisco logging :-(

The issue you are bumping into is that when these log lines are parsed by syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is no program name, or the
program name is the full %ASA-4-106023 part of the log line. There are so many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log lines.

Cisco can add a * to the date/time stamp to indicate that there is no time server configured on the device.
That makes the date/time invalid as far as parsing is concerned.

Cisco can add a period (.) to the date/time stamp to indicate that there is a time server configured on
the device, but the time server can not be reached. Again, this makes the date/time invalid as far as parsing
is concerned.

Cisco can add a sequence number at the start of the log line rather than starting the line with a date/6time stamp.
Invalid parsing again.

Cisco can leave out the sequence number but still include the trailing colon from the sequence number.

You get the idea. So many ways to get it wrong, and they never get it right :-(

Evan.

On 02/27/2018 04:46 PM, Tim Ghetti wrote:
Hi – having some trouble getting paterndb functional and looking for some help. I would like to use patterndb to parse my cisco ass firewall logs before sending it to elasticsearch. However when the messages get to elasticsearch, I don’t see the messages being parsed. Running pdbtool against the logs seems to work.

# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f /var/log/asatest.log |more
HOST=X.X.X.X
MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to OUTSIDE:X.X.X.X/X
PROGRAM=%ASA-6-305011
LEGACY_MSGHDR=%ASA-6-305011:
.classifier.class=system
.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b
Protocol=TCP
GlobalIP=X.X.X.X
GlobalPort=X
LocalIP=X.X.X.X
LocalPort=X
TAGS=.classifier.system



**********************
SYSLOG-NG CONF FILE
@version: 3.11
source s_network { tcp(); udp(); };
destination d_elastic {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://X.X.X.X:9200")
type("syslog")
flush-limit("1")
);
};
destination d_catchall { file("/var/log/catchall.log"); };
filter f_ciscoasa { host("X.X.X.X"); };
parser p_ciscoasa {db-parser(file("/etc/syslog-ng/patterndb.d/ciscoasa.pdb"));};
log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa); destination(d_elastic); flags(final, flow-control); };
log { source(s_network); destination(d_catchall); };


**********************
PATTERNDB FILE
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='4' pub_date='2018-02-19'>
<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>
<pattern>%ASA</pattern>
<description>
This ruleset covers the Cisco ASA firewalls
</description>
<rules>
<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86" class="system">
<!-- support: 1594 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system' provider='%ASA'>
<!-- support: 1369 -->
<patterns>
<pattern>Teardown dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown dynamic UDP translation from any:X.X.X.X/X to outside:X.X.X.X/X duration 0:00:00</test_message>
</example>
</examples>
</rule>
<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system' provider='%ASA'>
<!-- support: 1254 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101037<tel:(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X duration 0:00:00 bytes 132</test_message>
</example>
</examples>
</rule>
<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system' provider='%ASA'>
<!-- support: 1256 -->
<patterns>
<pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built outbound UDP connection 55101037<tel:(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to inside:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system' provider='%ASA'>
<!-- support: 1594 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection for faddr @IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort: @laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound ICMP connection for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system' provider='%ASA'>
<!-- support: 1763 -->
<patterns>
<pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built inbound UDP connection 55101078<tel:(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to identity:X.X.X.X/X (X.X.X.X/X)</test_message>
</example>
</examples>
</rule>
<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system' provider='%ASA'>
<!-- support: 1539 -->
<patterns>
<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for @ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4:DstIP:/@@ESTRING:DstPort: @***@ANYSTRING::@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Teardown UDP connection 55101084<tel:(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X duration 0:02:01 bytes 88</test_message>
</example>
</examples>
</rule>
<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system' provider='%ASA'>
<!-- support: 3648 -->
<patterns>
<pattern>Built dynamic @ESTRING:Protocol: @translation from @ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>
</patterns>
<examples>
<example>
<test_message program='%ASA'>Built dynamic TCP translation from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>
</example>
</examples>
</rule>
<rule provider='%ASA' class='system' id='39'>
<patterns>
<pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>
<pattern>regular translation creation failed for @ESTRING:: @src @ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code @NUMBER::@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>




Nￜnￜrￜￜￜￜ)emￜhￜyhiםￜw^ￜￜ



______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

[Scheidler, Balázs]

Can you give me a few inbound logs as received by syslog-ng? I would try to
fix up cisco parser that it works for your use-case, as it should.

Thanks

On Feb 28, 2018 21:48, "Tim Ghetti" <***@targetedsupport.com> wrote:

I tried using the cisco-parser and not having any luck with that either.
When I enable the parser, I actually stop seeing outbound traffic to
elasticsearch. My config is below





log {

source { udp(flags(no-parse)); };

parser { cisco-parser(); };

destination {

elasticsearch2(

client-mode("http")

cluster("ITESCL001")

index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")

cluster-url("http://192.168.101.199:9200
http://192.168.101.198:9200")

type("syslog")

flush-limit("1"));

};



}





[***@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv

tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size
262144 bytes

^C

0 packets captured

0 packets received by filter

0 packets dropped by kernel



*From:* syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] *On Behalf Of
*Scheidler, Balázs
*Sent:* Tuesday, February 27, 2018 11:06 PM
*To:* Syslog-ng users' and developers' mailing list <
syslog-***@lists.balabit.hu>

*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch



One additional feature you might be interested in is our cisco-parser()
feature.



It basically expects a flags(no-parse) input and then puts things in the
right buckets, while recognising the various formats that cisco uses.



It would be appreciated if you could check if it works with your setup and
I can help fixing it up if it doesn't.



The parser got added to recent syslog-ng versions, so you might need to
upgrade to a new version.



Bazsi



On Feb 27, 2018 22:29, "Tim Ghetti" <***@targetedsupport.com> wrote:

Thanks!! It sounds like this is part of my problem. If you would be so kind
as to send an example (or point me in the right direction) of how you are
changing the program name and message body for the cisco logs that would be
extremely helpful.



I also suspect that there may be another issue. I’m almost certain that my
parser should be picking up at least some of the logs but I am not seeing
any of the fields in ES showing up based on parsed logs. Assuming that
patterndb is working, would the below config be correct for getting
patterndb parsed variables to show as fields in ES?





destination d_elastic {

elasticsearch2(

client-mode("http")

cluster("ITESCL001")

index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")

cluster-url("http://X.X.X.X:9200")

type("syslog")

flush-limit("1")

);

};







*From:* syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] *On Behalf Of
*Evan Rempel
*Sent:* Tuesday, February 27, 2018 9:06 PM
*To:* syslog-***@lists.balabit.hu
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch



The program name will have an implied @ANYSTRING@ on the end, so if the
syslog payload could
be parsed to detect the program name of %ASA..... then your patterndb.xml
would work.
The problem is that they syslog payload can not be parsed.

What we do is use a hand crafted patterndb to detect all of the different
problem formats
that Cisco logs (in our environment) and change

a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the
leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.

All of the poor hosts, sequence numbers, timestamps etc are all thrown away.

After that, the internal buffers of syslog-ng contain a usable PROGRAM and
MESSAGE such that we can use
a patterndb to match the message part of the log line.

This does mean that we have two patterndb parsers for every log line but it
seems to work well for us.
What would work better is if Cisco would fix their logging, but that isn't
going to happen in my lifetime :-(

I hope that makes sense.

Evan.

On 02/27/2018 05:37 PM, Tim Ghetti wrote:

Ok, that makes sense. I sort of suspected that this was due to the program
name not matching, since pdbtool works when specifying the program name. Do
you know if there is a way to configure patterndb so that it matches the
program name, regardless of the full program name with event code? Is it
possible to regex the program or use the patterndb format i.e.
<pattern>%***@ANYSTRING::@</pattern> or something similar?



Tim



*From:* syslog-ng [mailto:syslog-ng-***@lists.balabit.hu
<syslog-ng-***@lists.balabit.hu>] *On Behalf Of *Evan Rempel
*Sent:* Tuesday, February 27, 2018 8:26 PM
*To:* syslog-***@lists.balabit.hu
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch



Welcome to the horrible world of Cisco logging :-(

The issue you are bumping into is that when these log lines are parsed by
syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is no
program name, or the
program name is the full %ASA-4-106023 part of the log line. There are so
many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log
lines.

Cisco can add a * to the date/time stamp to indicate that there is no time
server configured on the device.
That makes the date/time invalid as far as parsing is concerned.

Cisco can add a period (.) to the date/time stamp to indicate that there is
a time server configured on
the device, but the time server can not be reached. Again, this makes the
date/time invalid as far as parsing
is concerned.

Cisco can add a sequence number at the start of the log line rather than
starting the line with a date/6time stamp.
Invalid parsing again.

Cisco can leave out the sequence number but still include the trailing
colon from the sequence number.

You get the idea. So many ways to get it wrong, and they never get it right
:-(

Evan.

On 02/27/2018 04:46 PM, Tim Ghetti wrote:

Hi – having some trouble getting paterndb functional and looking for some
help. I would like to use patterndb to parse my cisco ass firewall logs
before sending it to elasticsearch. However when the messages get to
elasticsearch, I don’t see the messages being parsed. Running pdbtool
against the logs seems to work.



# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f
/var/log/asatest.log |more

HOST=X.X.X.X

MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to
OUTSIDE:X.X.X.X/X

PROGRAM=%ASA-6-305011

LEGACY_MSGHDR=%ASA-6-305011:

.classifier.class=system

.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b

Protocol=TCP

GlobalIP=X.X.X.X

GlobalPort=X

LocalIP=X.X.X.X

LocalPort=X

TAGS=.classifier.system







**********************

SYSLOG-NG CONF FILE

@version: 3.11

source s_network { tcp(); udp(); };

destination d_elastic {

elasticsearch2(

client-mode("http")

cluster("ITESCL001")

index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")

cluster-url("http://X.X.X.X:9200")

type("syslog")

flush-limit("1")

);

};

destination d_catchall { file("/var/log/catchall.log"); };

filter f_ciscoasa { host("X.X.X.X"); };

parser p_ciscoasa {db-parser(file("/etc/syslog-
ng/patterndb.d/ciscoasa.pdb"));};

log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa);
destination(d_elastic); flags(final, flow-control); };

log { source(s_network); destination(d_catchall); };





**********************

PATTERNDB FILE

<?xml version='1.0' encoding='UTF-8'?>

<patterndb version='4' pub_date='2018-02-19'>

<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>

<pattern>%ASA</pattern>

<description>

This ruleset covers the Cisco ASA firewalls

</description>

<rules>

<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86"
class="system">

<!-- support: 1594 -->

<patterns>

<pattern>Teardown @ESTRING:Protocol: @connection for faddr
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort:
@laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown ICMP connection for
faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>

</example>

</examples>

</rule>

<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system'
provider='%ASA'>

<!-- support: 1369 -->

<patterns>

<pattern>Teardown dynamic @ESTRING:Protocol: @translation from
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown dynamic UDP
translation from any:X.X.X.X/X to outside:X.X.X.X/X duration
0:00:00</test_message>

</example>

</examples>

</rule>

<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system'
provider='%ASA'>

<!-- support: 1254 -->

<patterns>

<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort:
@@ESTRING::@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown UDP connection
55101037 <(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X
duration 0:00:00 bytes 132</test_message>

</example>

</examples>

</rule>

<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system'
provider='%ASA'>

<!-- support: 1256 -->

<patterns>

<pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING::
@for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built outbound UDP connection
55101037 <(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to
inside:X.X.X.X/X (X.X.X.X/X)</test_message>

</example>

</examples>

</rule>

<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system'
provider='%ASA'>

<!-- support: 1594 -->

<patterns>

<pattern>Built inbound @ESTRING:Protocol: @connection for faddr
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort:
@laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built inbound ICMP connection
for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>

</example>

</examples>

</rule>

<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system'
provider='%ASA'>

<!-- support: 1763 -->

<patterns>

<pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING::
@for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to
identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built inbound UDP connection
55101078 <(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to
identity:X.X.X.X/X (X.X.X.X/X)</test_message>

</example>

</examples>

</rule>

<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system'
provider='%ASA'>

<!-- support: 1539 -->

<patterns>

<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to
identity:@IPv4:DstIP:/@@ESTRING:DstPort:
@***@ANYSTRING::@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown UDP connection
55101084 <(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X
duration 0:02:01 bytes 88</test_message>

</example>

</examples>

</rule>

<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system'
provider='%ASA'>

<!-- support: 3648 -->

<patterns>

<pattern>Built dynamic @ESTRING:Protocol: @translation from
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@
@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built dynamic TCP translation
from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>

</example>

</examples>

</rule>

<rule provider='%ASA' class='system' id='39'>

<patterns>

<pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@
@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>

<pattern>regular translation creation failed for @ESTRING:: @src
@ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code
@NUMBER::@</pattern>

</patterns>

</rule>

</rules>

</ruleset>

</patterndb>





Nￜnￜrￜￜￜￜ)emￜhￜyhiםￜw^ￜￜ




____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?
product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?
product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

[Scheidler, Balázs]

Hi,

Could you pls send a couple of messages? I would really like to make sure
this works out of the box.

Thanks

On Mar 1, 2018 16:08, "Scheidler, Balázs" <***@balabit.com>
wrote:

Can you give me a few inbound logs as received by syslog-ng? I would try to
fix up cisco parser that it works for your use-case, as it should.

Thanks

On Feb 28, 2018 21:48, "Tim Ghetti" <***@targetedsupport.com> wrote:

I tried using the cisco-parser and not having any luck with that either.
When I enable the parser, I actually stop seeing outbound traffic to
elasticsearch. My config is below





log {

source { udp(flags(no-parse)); };

parser { cisco-parser(); };

destination {

elasticsearch2(

client-mode("http")

cluster("ITESCL001")

index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")

cluster-url("http://192.168.101.199:9200
http://192.168.101.198:9200")

type("syslog")

flush-limit("1"));

};



}





[***@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv

tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size
262144 bytes

^C

0 packets captured

0 packets received by filter

0 packets dropped by kernel



*From:* syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] *On Behalf Of
*Scheidler, Balázs
*Sent:* Tuesday, February 27, 2018 11:06 PM
*To:* Syslog-ng users' and developers' mailing list <
syslog-***@lists.balabit.hu>

*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch



One additional feature you might be interested in is our cisco-parser()
feature.



It basically expects a flags(no-parse) input and then puts things in the
right buckets, while recognising the various formats that cisco uses.



It would be appreciated if you could check if it works with your setup and
I can help fixing it up if it doesn't.



The parser got added to recent syslog-ng versions, so you might need to
upgrade to a new version.



Bazsi



On Feb 27, 2018 22:29, "Tim Ghetti" <***@targetedsupport.com> wrote:

Thanks!! It sounds like this is part of my problem. If you would be so kind
as to send an example (or point me in the right direction) of how you are
changing the program name and message body for the cisco logs that would be
extremely helpful.



I also suspect that there may be another issue. I’m almost certain that my
parser should be picking up at least some of the logs but I am not seeing
any of the fields in ES showing up based on parsed logs. Assuming that
patterndb is working, would the below config be correct for getting
patterndb parsed variables to show as fields in ES?





destination d_elastic {

elasticsearch2(

client-mode("http")

cluster("ITESCL001")

index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")

cluster-url("http://X.X.X.X:9200")

type("syslog")

flush-limit("1")

);

};







*From:* syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] *On Behalf Of
*Evan Rempel
*Sent:* Tuesday, February 27, 2018 9:06 PM
*To:* syslog-***@lists.balabit.hu
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch



The program name will have an implied @ANYSTRING@ on the end, so if the
syslog payload could
be parsed to detect the program name of %ASA..... then your patterndb.xml
would work.
The problem is that they syslog payload can not be parsed.

What we do is use a hand crafted patterndb to detect all of the different
problem formats
that Cisco logs (in our environment) and change

a) the PROGRAM to be cisco_ASA or generally cisco_XXX where the XXX is the
leading characters of the %XXX-#-##### of the syslog body.
b) the MESSAGE to be %XXX-#-####... for the rest of the line.

All of the poor hosts, sequence numbers, timestamps etc are all thrown away.

After that, the internal buffers of syslog-ng contain a usable PROGRAM and
MESSAGE such that we can use
a patterndb to match the message part of the log line.

This does mean that we have two patterndb parsers for every log line but it
seems to work well for us.
What would work better is if Cisco would fix their logging, but that isn't
going to happen in my lifetime :-(

I hope that makes sense.

Evan.

On 02/27/2018 05:37 PM, Tim Ghetti wrote:

Ok, that makes sense. I sort of suspected that this was due to the program
name not matching, since pdbtool works when specifying the program name. Do
you know if there is a way to configure patterndb so that it matches the
program name, regardless of the full program name with event code? Is it
possible to regex the program or use the patterndb format i.e.
<pattern>%***@ANYSTRING::@</pattern> or something similar?



Tim



*From:* syslog-ng [mailto:syslog-ng-***@lists.balabit.hu
<syslog-ng-***@lists.balabit.hu>] *On Behalf Of *Evan Rempel
*Sent:* Tuesday, February 27, 2018 8:26 PM
*To:* syslog-***@lists.balabit.hu
*Subject:* Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch



Welcome to the horrible world of Cisco logging :-(

The issue you are bumping into is that when these log lines are parsed by
syslog-ng (or any
syslog daemon that does not specifically understand Cisco logs) there is no
program name, or the
program name is the full %ASA-4-106023 part of the log line. There are so
many ways that Cisco
can log incorrectly it is not possible to say without seeing your exact log
lines.

Cisco can add a * to the date/time stamp to indicate that there is no time
server configured on the device.
That makes the date/time invalid as far as parsing is concerned.

Cisco can add a period (.) to the date/time stamp to indicate that there is
a time server configured on
the device, but the time server can not be reached. Again, this makes the
date/time invalid as far as parsing
is concerned.

Cisco can add a sequence number at the start of the log line rather than
starting the line with a date/6time stamp.
Invalid parsing again.

Cisco can leave out the sequence number but still include the trailing
colon from the sequence number.

You get the idea. So many ways to get it wrong, and they never get it right
:-(

Evan.

On 02/27/2018 04:46 PM, Tim Ghetti wrote:

Hi – having some trouble getting paterndb functional and looking for some
help. I would like to use patterndb to parse my cisco ass firewall logs
before sending it to elasticsearch. However when the messages get to
elasticsearch, I don’t see the messages being parsed. Running pdbtool
against the logs seems to work.



# pdbtool match -p /etc/syslog-ng/patterndb.d/ciscoasa.pdb -P %ASA -f
/var/log/asatest.log |more

HOST=X.X.X.X

MESSAGE=Built dynamic TCP translation from INSIDE:X.X.X.X/X to
OUTSIDE:X.X.X.X/X

PROGRAM=%ASA-6-305011

LEGACY_MSGHDR=%ASA-6-305011:

.classifier.class=system

.classifier.rule_id=e075efdc-c25f-5e49-a208-7661e3b5a29b

Protocol=TCP

GlobalIP=X.X.X.X

GlobalPort=X

LocalIP=X.X.X.X

LocalPort=X

TAGS=.classifier.system







**********************

SYSLOG-NG CONF FILE

@version: 3.11

source s_network { tcp(); udp(); };

destination d_elastic {

elasticsearch2(

client-mode("http")

cluster("ITESCL001")

index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")

cluster-url("http://X.X.X.X:9200")

type("syslog")

flush-limit("1")

);

};

destination d_catchall { file("/var/log/catchall.log"); };

filter f_ciscoasa { host("X.X.X.X"); };

parser p_ciscoasa {db-parser(file("/etc/syslog-n
g/patterndb.d/ciscoasa.pdb"));};

log { source(s_network); filter(f_ciscoasa); parser(p_ciscoasa);
destination(d_elastic); flags(final, flow-control); };

log { source(s_network); destination(d_catchall); };





**********************

PATTERNDB FILE

<?xml version='1.0' encoding='UTF-8'?>

<patterndb version='4' pub_date='2018-02-19'>

<ruleset name='%ASA' id='a300d776-8bd7-834d-a4a9-23eb81a4b3ba'>

<pattern>%ASA</pattern>

<description>

This ruleset covers the Cisco ASA firewalls

</description>

<rules>

<rule provider="%ASA" id="b3de7699-8213-c744-944e-9413298afe86"
class="system">

<!-- support: 1594 -->

<patterns>

<pattern>Teardown @ESTRING:Protocol: @connection for faddr
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort:
@laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown ICMP connection for
faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>

</example>

</examples>

</rule>

<rule id='90d0f8c9-7591-d44e-b886-2f7e5cb17ce6' class='system'
provider='%ASA'>

<!-- support: 1369 -->

<patterns>

<pattern>Teardown dynamic @ESTRING:Protocol: @translation from
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@
@IPv4:GlobalIP:/@@ESTRING:GlobalPort: @***@ANYSTRING::@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown dynamic UDP
translation from any:X.X.X.X/X to outside:X.X.X.X/X duration
0:00:00</test_message>

</example>

</examples>

</rule>

<rule id='8f0a8d57-80c6-4745-8a8a-5ce018bb0d87' class='system'
provider='%ASA'>

<!-- support: 1254 -->

<patterns>

<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @to @ESTRING:::@
@IPv4:SrcIP:/@@ESTRING:SrcPort: @@ESTRING::@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown UDP connection
55101037 <(55)%20101%20037> for outside:X.X.X.X/X to inside:X.X.X.X/X
duration 0:00:00 bytes 132</test_message>

</example>

</examples>

</rule>

<rule id='00c0732d-1e34-7340-a75f-21198bf71137' class='system'
provider='%ASA'>

<!-- support: 1256 -->

<patterns>

<pattern>Built outbound @ESTRING:Protocol: @connection @ESTRING::
@for @ESTRING:::@@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@ to
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built outbound UDP connection
55101037 <(55)%20101%20037> for outside:X.X.X.X/X (X.X.X.X/X) to
inside:X.X.X.X/X (X.X.X.X/X)</test_message>

</example>

</examples>

</rule>

<rule id='4a586711-ebe2-dc4d-bf6e-e512666d8c5d' class='system'
provider='%ASA'>

<!-- support: 1594 -->

<patterns>

<pattern>Built inbound @ESTRING:Protocol: @connection for faddr
@IPv4:SrcIP:/@@ESTRING:SrcPort: @gaddr @IPv4:GlobalIP:/@@ESTRING:GlobalPort:
@laddr @IPv4:LocalIP:/@@ESTRING:LocalPort:@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built inbound ICMP connection
for faddr X.X.X.X/X gaddr X.X.X.X/X laddr X.X.X.X/X</test_message>

</example>

</examples>

</rule>

<rule id='8be7928d-66e7-7042-abd5-869d6b49c56e' class='system'
provider='%ASA'>

<!-- support: 1763 -->

<patterns>

<pattern>Built inbound @ESTRING:Protocol: @connection @ESTRING::
@for @ESTRING::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @(@ESTRING::)@ to
identity:@IPv4:DstIP:/@@ESTRING:DstPort: @(@ESTRING::)@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built inbound UDP connection
55101078 <(55)%20101%20078> for inside:X.X.X.X/X (X.X.X.X/X) to
identity:X.X.X.X/X (X.X.X.X/X)</test_message>

</example>

</examples>

</rule>

<rule id='20aee256-b4f0-8b4d-93cb-263d5338fd21' class='system'
provider='%ASA'>

<!-- support: 1539 -->

<patterns>

<pattern>Teardown @ESTRING:Protocol: @connection @ESTRING:: @for
@ESTRING:::@@IPv4:SrcIP:/@@ESTRING:SrcPort: @to identity:@IPv4
:DstIP:/@@ESTRING:DstPort: @***@ANYSTRING::@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Teardown UDP connection
55101084 <(55)%20101%20084> for inside:X.X.X.X/X to identity:X.X.X.X/X
duration 0:02:01 bytes 88</test_message>

</example>

</examples>

</rule>

<rule id='e075efdc-c25f-5e49-a208-7661e3b5a29b' class='system'
provider='%ASA'>

<!-- support: 3648 -->

<patterns>

<pattern>Built dynamic @ESTRING:Protocol: @translation from
@ESTRING:::@@IPv4:LocalIP:/@@ESTRING:LocalPort: @to @ESTRING:::@
@IPv4:GlobalIP:/@@ESTRING:GlobalPort:@</pattern>

</patterns>

<examples>

<example>

<test_message program='%ASA'>Built dynamic TCP translation
from any:X.X.X.X/X to outside:X.X.X.X/X</test_message>

</example>

</examples>

</rule>

<rule provider='%ASA' class='system' id='39'>

<patterns>

<pattern>Cleared @ESTRING:: @urgent flag from @ESTRING:::@
@ESTRING::/@@NUMBER::@ to @ESTRING: ::@@ESTRING::/@@NUMBER::@</pattern>

<pattern>regular translation creation failed for @ESTRING:: @src
@ESTRING:::@@ESTRING:: @dst @ESTRING: ::@@ESTRING:: @(type @NUMBER::@, code
@NUMBER::@</pattern>

</patterns>

</rule>

</rules>

</ruleset>

</patterndb>





Nￜnￜrￜￜￜￜ)emￜhￜyhiםￜw^ￜￜ




____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=
syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=
syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

[Tim Ghetti]

Hi

I was able to resolve some of my issues. The first thing I did was stop all logging to ES and delete all my data. Once I started logging to ES again with a json template statement in the output rule I saw that the logs were in fact being parsed by patterndb. Also I noticed that using the “logging device-id” statement on the ASA sends over an empty PROGRAM field in syslog – so I removed that statement. Now I have patterndb working correctly and I just have to go through and fine tune my parser rules.

I am interested in why the cisco-parser() statement would not send any output. Below are some log samples. Let me know if there is anything helpful here. Thanks for everyone’s help!


Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-106015: Deny TCP (no connection) from XXX.XXX.192.57/1147 to XXX.XXX.105.15/445 flags RST ACK on interface OUTSIDE
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-305011: Built dynamic TCP translation from OUTSIDE:XXX.XXX.192.57/2004(LOCAL\user) to OUTSIDE:69.147.160.165/2004
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-302013: Built inbound TCP connection 787029 for OUTSIDE:XXX.XXX.192.57/2004 (XXX.XXX.160.165/2004)(LOCAL\user) to OUTSIDE:XXX.XXX.165.71/443 (XXX.XXX.165.71/443) (user)
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-302016: Teardown UDP connection 786540 for OUTSIDE:XXX.XXX.4.101/123 to INSIDE:XXX.XXX.105.61/123 duration 0:02:02 bytes 48
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-305011: Built dynamic TCP translation from OUTSIDE:XXX.XXX.192.57/2004(LOCAL\user) to OUTSIDE: XXX.XXX.160.165/2004
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-302014: Teardown TCP connection 787027 for OUTSIDE: XXX.XXX.44.11/443 to INSIDE:XXX.XXX.105.122/50330 duration 0:00:00 bytes 25227 TCP FINs from INSIDE
Feb 25 22:14:13.247139 XXX.XXX.31.1 %ASA-7-609001: Built local-host outside: XXX.XXX.224.196
Feb 25 22:14:13.247139 XXX.XXX.31.1 %ASA-6-302020: Built inbound ICMP connection for faddr XXX.XXX.224.196/54900 gaddr XXX.XXX.77.81/0 laddr XXX.XXX.77.81/0
Feb 25 22:14:13.247349 XXX.XXX.31.1 %ASA-6-302021: Teardown ICMP connection for faddr XXX.XXX.224.196/54900 gaddr XXX.XXX.77.81/0 laddr XXX.XXX.77.81/0
Feb 25 22:14:13.257785 XXX.XXX.159.2 %ASA-6-302014: Teardown TCP connection 369356 for OUTSIDE: XXX.XXX.167.21/443 to INSIDE:XXX.XXX.135.46/58914 duration 0:00:18 bytes 7264 TCP Reset-O from OUTSIDE
Feb 25 22:14:13.257885 XXX.XXX.159.2 %ASA-6-305012: Teardown dynamic TCP translation from INSIDE:XXX.XXX.135.46/58914 to OUTSIDE: XXX.XXX.45.90/58914 duration 0:00:18
Feb 25 22:14:13.261045 XXX.XXX.159.2 %ASA-6-305011: Built dynamic TCP translation from INSIDE:XXX.XXX.135.46/50748 to OUTSIDE: XXX.XXX.45.90/50748
Feb 25 22:14:13.261145 XXX.XXX.159.2 %ASA-6-302013: Built outbound TCP connection 369392 for OUTSIDE: XXX.XXX.167.21/443 (XXX.XXX.167.21/443) to INSIDE:XXX.XXX.135.46/50748 (XXX.XXX.45.90/50748)
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-302014: Teardown TCP connection 633281 for OUTSIDE: XXX.XXX.44.11/443 to INSIDE:XXX.XXX.105.122/50330 duration 0:00:00 bytes 0 Failover primary closed
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-302014: Teardown TCP connection 726445 for OUTSIDE:XXX.XXX.192.57/1074(LOCAL\user) to INSIDE:XXX.XXX.101.104/443 duration 2:13:31 bytes 58884 TCP Reset-O from OUTSIDE (user)
Feb 26 03:14:13.000000 BHD-MDC-FW1 : %ASA-6-305011: Built dynamic TCP translation from OUTSIDE:XXX.XXX.192.57/2005(LOCAL\user) to OUTSIDE: XXX.XXX.160.165/2005

From: syslog-ng [mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Scheidler, Balázs
Sent: Monday, March 5, 2018 7:55 AM
To: Syslog-ng users' and developers' mailing list <syslog-***@lists.balabit.hu<mailto:syslog-***@lists.balabit.hu>>
Subject: Re: [syslog-ng] Cisco ASA parsing with patterndb/elasticsearch

Hi,

Could you pls send a couple of messages? I would really like to make sure this works out of the box.

Thanks

On Mar 1, 2018 16:08, "Scheidler, Balázs" <***@balabit.com<mailto:***@balabit.com>> wrote:
Can you give me a few inbound logs as received by syslog-ng? I would try to fix up cisco parser that it works for your use-case, as it should.

Thanks

On Feb 28, 2018 21:48, "Tim Ghetti" <***@targetedsupport.com<mailto:***@targetedsupport.com>> wrote:
I tried using the cisco-parser and not having any luck with that either. When I enable the parser, I actually stop seeing outbound traffic to elasticsearch. My config is below


log {
source { udp(flags(no-parse)); };
parser { cisco-parser(); };
destination {
elasticsearch2(
client-mode("http")
cluster("ITESCL001")
index("logstash-syslogng_${YEAR}.${MONTH}.${DAY}")
cluster-url("http://192.168.101.199:9200 http://192.168.101.198:9200")
type("syslog")
flush-limit("1"));
};

}


[***@ITLOG001 conf.d]# tcpdump -nnSXi ens192 port 9200 -vv
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel