[syslog-ng] How to convert syslog-ng logs into xml format

[Padmanabhan, Rajeesh (GE Healthcare)]

Hi All,

I need to convert syslog-ng logs into xml format. Could you please help
me to do the same. If someone can send me a sample format, would be
great.

Thanks in advance.

-Rajeesh

[Alexander Clouter]

Hi,

Post by Padmanabhan, Rajeesh (GE Healthcare)
Hi All,
I need to convert syslog-ng logs into xml format. Could you please help
me to do the same. If someone can send me a sample format, would be
great.

Oh dear...sounds like someone caught 'xml fever', it happens to all of us at
some stage and we usually recover from the illness when its too late and the
project is already using it...

XML is something thats abused more often than its used properly. Its useful
if you have two rather different systems that have no common data exchange
mechanism between them and so XML is used as a 'common' format. Bear in mind
that this means is at both ends you have to get programming tech-monkeys to
write scripts to convert the data to XML and then back from XML into another
format. In situations like this you do not care that it could take two weeks
of solid CPU crunching to import the data, just as long as it gets there...

Now as you have not given much detail about why you need the files in an XML
format or what you plan on doing with the files the follow format is probably
'suitable':

<?xml version="1.0" encoding="UTF-8"?>
<syslog>
<message>first syslog message</message>
<message>second syslog message</message>
...
</syslog>

I'm finding it hard to think of a situation where XML and syslog data would
be a good idea given that its rather easy to parse and store syslog data
already. People want to archive and easily search (usually with nothing more
than grep) their syslog logs and in an XML format searching your logs *will*
slow down by at least a factor of 1000.

A far more suitable format, if flat text files is not good enough, is a SQL
database; probably the only real useful application of an SQL database[1].

I'm unsure how to help you recover from your XML fever, it was difficult for
me to grow out of. I learn from my mistakes and unfortunately for me when
you base a project around XML the mistakes (for me at least) do not have an
effect till much later on. There is nothing on the Internet as suggested
reading but I am happy to discuss this off list if you want to.

I did some digging around to try to see if there was anything commercial
doing this and found[2] something and was shocked at how they were doing
it...its a bad schema and seems to be a solution to something that is not
actually a problem.

*Everything* that deals with logging data should support syslog built
logfiles (you might need to use syslog-ng with its template() command to
structure the logfile to be in a format that the import engine expects
though), if it does not you should look for some other software. My gut
feelings tell me this is an internal homebrew project though..

I hope the 'fever' breaks soon :)

Cheers

Alex

[1] thats another rant of mine though :)
[2] http://www.kiwisyslog.com/kb/idx/4/125/article/

[Padmanabhan, Rajeesh (GE Healthcare)]

Hi Alex,

Thanks a lot for your quick response. Infact my role here to act as
'Tech-monkey' :)
I have a syslog-ng clients & server. Clients should be able to transfer
logs in xml format to server. & server stores in a local location(xml
file). Idea is to upload this to web & backup, which can read data in
xml fomat.
Could you please explain me to create a template that generate logs in
xml format.

Cheers!
Rajeesh

-----Original Message-----
From: syslog-ng-***@lists.balabit.hu
[mailto:syslog-ng-***@lists.balabit.hu] On Behalf Of Alexander
Clouter
Sent: Sunday, February 11, 2007 4:21 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] How to convert syslog-ng logs into xml format

Hi,

Post by Padmanabhan, Rajeesh (GE Healthcare)
Hi All,
I need to convert syslog-ng logs into xml format. Could you please
help me to do the same. If someone can send me a sample format, would
be great.

Oh dear...sounds like someone caught 'xml fever', it happens to all of
us at some stage and we usually recover from the illness when its too
late and the project is already using it...

XML is something thats abused more often than its used properly. Its
useful if you have two rather different systems that have no common data
exchange mechanism between them and so XML is used as a 'common' format.
Bear in mind that this means is at both ends you have to get programming
tech-monkeys to write scripts to convert the data to XML and then back
from XML into another format. In situations like this you do not care
that it could take two weeks of solid CPU crunching to import the data,
just as long as it gets there...

Now as you have not given much detail about why you need the files in an
XML format or what you plan on doing with the files the follow format is
probably
'suitable':

<?xml version="1.0" encoding="UTF-8"?>
<syslog>
<message>first syslog message</message>
<message>second syslog message</message>
...
</syslog>

I'm finding it hard to think of a situation where XML and syslog data
would be a good idea given that its rather easy to parse and store
syslog data already. People want to archive and easily search (usually
with nothing more than grep) their syslog logs and in an XML format
searching your logs *will* slow down by at least a factor of 1000.

A far more suitable format, if flat text files is not good enough, is a
SQL database; probably the only real useful application of an SQL
database[1].

I'm unsure how to help you recover from your XML fever, it was difficult
for me to grow out of. I learn from my mistakes and unfortunately for
me when you base a project around XML the mistakes (for me at least) do
not have an effect till much later on. There is nothing on the Internet
as suggested reading but I am happy to discuss this off list if you want
to.

I did some digging around to try to see if there was anything commercial
doing this and found[2] something and was shocked at how they were doing
it...its a bad schema and seems to be a solution to something that is
not actually a problem.

*Everything* that deals with logging data should support syslog built
logfiles (you might need to use syslog-ng with its template() command to
structure the logfile to be in a format that the import engine expects
though), if it does not you should look for some other software. My gut
feelings tell me this is an internal homebrew project though..

I hope the 'fever' breaks soon :)

Cheers

Alex

[1] thats another rant of mine though :) [2]
http://www.kiwisyslog.com/kb/idx/4/125/article/

_______________________________________________
syslog-ng maillist - syslog-***@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

[Evan Rempel]

One of the chalenges you will face with XML and syslog is that XML can not be validated until
all of the input has been XML parsed. If you have a 1GB XML file, your XML parser will probably
read the entire thing into RAM. Perhaps that works for you, but with our 4GB log file, the XML
parsing and internal data storage overhead, this could easily consume 8GB of RAM.

When you indicate that you need to put this on the WEB, you will find that XML of large data
will bring your web server to its knees. Parsing through 1GB of data to return a line or two
will be very CPU intensive, and on a clickable (web) setup, this will be repeated often.

I also think you someone at your site has a fever, and I hope they seek some prfessional
advice rather than self-diagnosing the issue :-)

Evan.

Post by Padmanabhan, Rajeesh (GE Healthcare)
Hi Alex,
Thanks a lot for your quick response. Infact my role here to act as
'Tech-monkey' :)
I have a syslog-ng clients & server. Clients should be able to transfer
logs in xml format to server. & server stores in a local location(xml
file). Idea is to upload this to web & backup, which can read data in
xml fomat.
Could you please explain me to create a template that generate logs in
xml format.
Cheers!
Rajeesh
-----Original Message-----
Clouter
Sent: Sunday, February 11, 2007 4:21 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] How to convert syslog-ng logs into xml format
Hi,

Post by Padmanabhan, Rajeesh (GE Healthcare)
Hi All,
I need to convert syslog-ng logs into xml format. Could you please
help me to do the same. If someone can send me a sample format, would
be great.

Oh dear...sounds like someone caught 'xml fever', it happens to all of
us at some stage and we usually recover from the illness when its too
late and the project is already using it...
XML is something thats abused more often than its used properly. Its
useful if you have two rather different systems that have no common data
exchange mechanism between them and so XML is used as a 'common' format.
Bear in mind that this means is at both ends you have to get programming
tech-monkeys to write scripts to convert the data to XML and then back
from XML into another format. In situations like this you do not care
that it could take two weeks of solid CPU crunching to import the data,
just as long as it gets there...
Now as you have not given much detail about why you need the files in an
XML format or what you plan on doing with the files the follow format is
probably
<?xml version="1.0" encoding="UTF-8"?>
<syslog>
<message>first syslog message</message>
<message>second syslog message</message>
...
</syslog>
I'm finding it hard to think of a situation where XML and syslog data
would be a good idea given that its rather easy to parse and store
syslog data already. People want to archive and easily search (usually
with nothing more than grep) their syslog logs and in an XML format
searching your logs *will* slow down by at least a factor of 1000.
A far more suitable format, if flat text files is not good enough, is a
SQL database; probably the only real useful application of an SQL
database[1].
I'm unsure how to help you recover from your XML fever, it was difficult
for me to grow out of. I learn from my mistakes and unfortunately for
me when you base a project around XML the mistakes (for me at least) do
not have an effect till much later on. There is nothing on the Internet
as suggested reading but I am happy to discuss this off list if you want
to.
I did some digging around to try to see if there was anything commercial
doing this and found[2] something and was shocked at how they were doing
it...its a bad schema and seems to be a solution to something that is
not actually a problem.
*Everything* that deals with logging data should support syslog built
logfiles (you might need to use syslog-ng with its template() command to
structure the logfile to be in a format that the import engine expects
though), if it does not you should look for some other software. My gut
feelings tell me this is an internal homebrew project though..
I hope the 'fever' breaks soon :)
Cheers
Alex
[1] thats another rant of mine though :) [2]
http://www.kiwisyslog.com/kb/idx/4/125/article/
_______________________________________________
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

[Alexander Clouter]

Hi,

Post by Padmanabhan, Rajeesh (GE Healthcare)
I have a syslog-ng clients & server. Clients should be able to transfer
logs in xml format to server. & server stores in a local location(xml
file). Idea is to upload this to web & backup, which can read data in
xml fomat.

I think you might have missed the key point behind my reply...you still are
yet to give a reason why you need it in an XML format. If its for
'archiving' and 'retrieval' then you have just given two reasons *not* to use
XML. As I mentioned, XML best suited for exporting and importing data[1] and
not much else...you want live parsing of data and XML is the last thing you
want to be using.

Archiving logs is best done in raw flat text files, to make the data suitable
for access via a web frontend you should consider some like pre-processing[1]
the data and dumping it into an SQL database. This is the perfect storage
format for accessing the data.

Meanwhile you archive with flat textfiles. To restore the data you simply
replay it back through the SQL pre-processor thingy.

The way, when I get around to doing it, is to use SQL triggers to
post-process the pre-processed data. So the 'pre-processing' stage would
involve breaking it out into columns the main information[2] and then a
post-processing trigger would (for example) pull out the SMTP message ID for
an SMTP transaction and the to/from fields to dump things into an SMTP
specific table.

Seriously, you have not given a reason to use XML yet and I strongly
recommend you go back to the drawing board and have a rethink.

Do not feel bad about this, we all make this mistake, we are trying to help
you to avoid it :)

Cheers

Alex

[1] break out the timestamp, host IP, process, pid, log level, etc into
separate columns
[2] there are already piping scripts available for syslog-ng out there to do
this

Post by Padmanabhan, Rajeesh (GE Healthcare)
Could you please explain me to create a template that generate logs in
xml format.
Cheers!
Rajeesh
-----Original Message-----
Clouter
Sent: Sunday, February 11, 2007 4:21 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] How to convert syslog-ng logs into xml format
Hi,

Post by Padmanabhan, Rajeesh (GE Healthcare)
Hi All,
I need to convert syslog-ng logs into xml format. Could you please
help me to do the same. If someone can send me a sample format, would
be great.

Oh dear...sounds like someone caught 'xml fever', it happens to all of
us at some stage and we usually recover from the illness when its too
late and the project is already using it...
XML is something thats abused more often than its used properly. Its
useful if you have two rather different systems that have no common data
exchange mechanism between them and so XML is used as a 'common' format.
Bear in mind that this means is at both ends you have to get programming
tech-monkeys to write scripts to convert the data to XML and then back
from XML into another format. In situations like this you do not care
that it could take two weeks of solid CPU crunching to import the data,
just as long as it gets there...
Now as you have not given much detail about why you need the files in an
XML format or what you plan on doing with the files the follow format is
probably
<?xml version="1.0" encoding="UTF-8"?>
<syslog>
<message>first syslog message</message>
<message>second syslog message</message>
...
</syslog>
I'm finding it hard to think of a situation where XML and syslog data
would be a good idea given that its rather easy to parse and store
syslog data already. People want to archive and easily search (usually
with nothing more than grep) their syslog logs and in an XML format
searching your logs *will* slow down by at least a factor of 1000.
A far more suitable format, if flat text files is not good enough, is a
SQL database; probably the only real useful application of an SQL
database[1].
I'm unsure how to help you recover from your XML fever, it was difficult
for me to grow out of. I learn from my mistakes and unfortunately for
me when you base a project around XML the mistakes (for me at least) do
not have an effect till much later on. There is nothing on the Internet
as suggested reading but I am happy to discuss this off list if you want
to.
I did some digging around to try to see if there was anything commercial
doing this and found[2] something and was shocked at how they were doing
it...its a bad schema and seems to be a solution to something that is
not actually a problem.
*Everything* that deals with logging data should support syslog built
logfiles (you might need to use syslog-ng with its template() command to
structure the logfile to be in a format that the import engine expects
though), if it does not you should look for some other software. My gut
feelings tell me this is an internal homebrew project though..
I hope the 'fever' breaks soon :)
Cheers
Alex
[1] thats another rant of mine though :) [2]
http://www.kiwisyslog.com/kb/idx/4/125/article/
_______________________________________________
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
_______________________________________________
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html