syslog-ng+loganalyzer搭建日志集中监控平台

2017年1月11日 发表评论 阅读评论

这里还是承接同事的需求,想要将所有网络设备的日志汇总后在一个平台上进行展示。在上一篇syslog-ng日志应用详解中提到了,通过syslog-ng创建日志集中服务器。可以通过syslog-ng+loganalyzer实现将接受采集过来的日志入库并在web页面上统一展示。

一、数据库配置

1、创建数据库及表结构

mysql> CREATE DATABASE Syslog character set utf8;
mysql> USE Syslog;
mysql> CREATE TABLE SystemEvents
(
        ID int unsigned not null auto_increment primary key,
        CustomerID bigint,
        ReceivedAt datetime NULL,
        DeviceReportedTime datetime NULL,
        Facility smallint NULL,
        Priority smallint NULL,
        FromHost varchar(60) NULL,
        Message text,
        NTSeverity int NULL,
        Importance int NULL,
        EventSource varchar(60),
        EventUser varchar(60) NULL,
        EventCategory int NULL,
        EventID int NULL,
        EventBinaryData text NULL,
        MaxAvailable int NULL,
        CurrUsage int NULL,
        MinUsage int NULL,
        MaxUsage int NULL,
        InfoUnitID int NULL ,
        SysLogTag varchar(60),
        EventLogType varchar(60),
        GenericFileName VarChar(60),
        SystemID int NULL
);
mysql> CREATE TABLE SystemEventsProperties
 (
         ID int unsigned not null auto_increment primary key,
         SystemEventID int NULL ,
         ParamName varchar(255) NULL ,
         ParamValue text NULL
 ); 

创建表结构也有不同的,国外一个站点上看到有人按如下结构创建:

CREATE TABLE `logs` (
    `host` varchar(32) DEFAULT NULL,
    `facility` varchar(10) DEFAULT NULL,
    `priority` varchar(10) DEFAULT NULL,
    `level` varchar(10) DEFAULT NULL,
    `tag` varchar(10) DEFAULT NULL,
    `datetime` datetime DEFAULT NULL,
    `program` varchar(15) DEFAULT NULL,
    `msg` text,
    `seq` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    PRIMARY KEY (`seq`),
    KEY `host` (`host`),
    KEY `program` (`program`),
    KEY `datetime` (`datetime`),
    KEY `priority` (`priority`),
    KEY `facility` (`facility`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

2、设置数据库权限

mysql> GRANT ALL ON Syslog.* TO syslog_ng@localhost IDENTIFIED BY 'syslog_ngpass';
mysql> FLUSH PRIVILEGES; 

二、syslog-ng.conf 配置

source s_remote {
         tcp(ip(0.0.0.0) port(514));
         udp(ip(0.0.0.0) port(514));
};
destination d_mysql {
sql(type(mysql)
host("localhost") username("syslog_ng") password("syslog_ngpass")
database("Syslog") table("SystemEvents")
columns("ID int unsigned not null auto_increment primary key","ReceivedAt datetime NULL", "DeviceReportedTime datetime NULL",
"Facility smallint NULL","Priority smallint NULL","FromHost varchar(60) NULL",
"Message text","InfoUnitID int NULL","SysLogTag varchar(60)",
"CustomerID bigint","NTSeverity int NULL","Importance int NULL","EventSource varchar(60)","EventUser varchar(60) NULL",
"EventCategory int NULL","EventID int NULL","EventBinaryData text NULL","MaxAvailable int NULL","CurrUsage int NULL","MinUsage int NULL",
"MaxUsage int NULL","EventLogType varchar(60)","GenericFileName VarChar(60)","SystemID int NULL")
values("","$R_ISODATE", "$S_ISODATE","$FACILITY_NUM","$LEVEL_NUM","$HOST",
"$MSGONLY","1","$MSGHDR","","","","","","","","","","","","","","","")
indexes("ID","ReceivedAt","Facility","Priority","FromHost","SysLogTag",));
};
log { source(s_remote); destination(d_mysql); };

三、loganalyzer配置

到 http://loganalyzer.adiscon.com/ 页面下载最新的loganalyzer程序,并放到apache的根目录下 。

cd  loganalyzer-*
mkdir /var/www/html/log
mv ./src/*  /var/www/html/log
cp contrib/* /var/www/html/log
cd /var/www/html/loganalyzersh
sh  configure.sh  

配置完成后,在浏览器中输入http://ip/log/install.php 进行安装即可,都是下一步下一步的操作,没有什么可搞性 。这个上个搞完后的效果图:

loganalyzer

四、日志清理

数据库随着日志的堆积会越来越大,这样会导致在前端页面查询会变慢,所以再搞个清理任务,30天以前的日志进行每天定时清理。

cat >/etc/cron.daily/syslog-clean.sh <<EOF
#!/bin/bash
MYSQL_USER="syslog_ng"
MYSQL_PASS="syslog_ngpass"
MYSQL_DB="Syslog"
mysql  -u\${MYSQL_USER} -p\${MYSQL_PASS} \${MYSQL_DB} -e "DELETE FROM SystemEvents WHERE ReceivedAt < DATE_SUB(CURDATE(),INTERVAL 30 DAY)"
EOF
chmod 700 /etc/cron.daily/syslog-clean.sh 

五、rsyslog与evtsys

1、rsyslog与loganalyzer的结合

由于syslog-ng语法结构比较灵活,所以这里选的和syslog-ng进行的集成,实际上loganalyzer也可以与rsyslog结合,具体的操作步骤如下:

安装包:
yum install rsyslog rsyslog-mysql mysql mysql-devel mysql-server php php-mysql php-pdo php-common php-gd httpd
导入rsyslog数据库:
mysql -u root -p < $(rpm -ql rsyslog-mysql | grep sql$)
数据库用户创建:
mysql -u root -p
mysql> grant all privileges on Syslog.* to logger@localhost identified by 'logger';
mysql> flush privileges;
mysql> exit; 

编辑rsyslog.conf文件,增加如下内容:

$ModLoad ommysql
*.* :ommysql:127.0.0.1,Syslog,logger,logger
$ModLoad imudp.so
$UDPServerRun 514 

其他如服务重启及loganalyzer安装步骤略过或同上。

2、evtsys

网络设备和LINUX对与syslog协议天然具有兼容性,只需要在配置中简单的指定,即可让syslog服务器接收其他主机的日志信息。windows通过evtsys也可以实现将自身的日志发送到日志服务器上去,操作很简单,下载evtsys http://code.google.com/p/eventlog-to-syslog/ 解压缩放到 C:\Windows\System32 ,cmd下执行如下操作:

evtsys -i -s 10 -h log-server-ip -p 514
net start evtsys 

参考页面:

https://anton.dollmaier.name/syslog-host-mit-syslog-ng-und-mysql

http://gm100861.blog.51cto.com/1930562/1191164




本站的发展离不开您的资助,金额随意,欢迎来赏!

You can donate through PayPal.
My paypal id: itybku@139.com
Paypal page: https://www.paypal.me/361way

  1. 本文目前尚无任何评论.
  1. 本文目前尚无任何 trackbacks 和 pingbacks.