Logstash收集json格式日志文件如何写配置文件

云计算 waitig 1351℃ 百度已收录 0评论

1、日志格式

{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}

我们如果收集这个日志只是做简单的配置。如下:

input {
    file {

        path => ["/home/elk/logstash-5.6.3/request"]
        type => "chenxun"
    }
}


output {

    stdout {
        codec => rubydebug
    }

    elasticsearch {
        hosts => "192.168.2.181:9200"

    }
}

那么收集到的结果是:

{
    "_index": "logstash-2017.11.22",
    "_type": "chenxun",
    "_id": "AV_iTR0AM1H1mf2je0nC",
    "_version": 1,
    "_score": 1,
    "_source": {
        "@version": "1",
        "host": "Ubuntu-20170424",
        "path": "/home/elk/logstash-5.6.3/request",
        "@timestamp": "2017-11-22T05:57:05.383Z",
        "message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",
        "type": "chenxun"
    }
}

即会将json记录做为一个字符串放到”message”下,这不是我们想要的结果,是让logstash自动解析json记录,将各字段放入elasticsearch中。下面介绍如何配置.

1.直接设置codec => json

input {
    file {

        path => ["/home/elk/logstash-5.6.3/request"]
        type => "chenxun"
        codec => json

    }   
}

这个时候看看结果: 已经把json解析到各个字段中去了

{
    "_index": "logstash-2017.11.22",
    "_type": "136587",
    "_id": "AV_iXHbGM1H1mf2jfF4d",
    "_version": 1,
    "_score": 1,
    "_source": {
        "48": "136587",
        "10018": 0,
        "10070": 0,
        "10071": 0,
        "10190": 0,
        "10191": 0,
        "path": "/home/elk/logstash-5.6.3/request",
        "@timestamp": "2017-11-22T06:13:51.361Z",
        "@version": "1",
        "host": "Ubuntu-20170424",
        "type": "136587"
    }
}

可以设置编码格式:(收集中文日志)

codec => json {
            charset => "UTF-8"
        }

2、使用filter json

配置如下:

input {
    file {

        path => ["/home/elk/logstash-5.6.3/request"]

    }
}

filter {
        json {
            source => "message"
            #target => "doc"
            #remove_field => ["message"]
        }        
}

output {

    stdout {
        codec => rubydebug
    }

    elasticsearch {
        hosts => "192.168.2.181:9200"

    }
}

输入结果:

{
    "_index": "logstash-2017.11.22",
    "_type": "136587",
    "_id": "AV_igupKM1H1mf2jfxm2",
    "_version": 1,
    "_score": 1,
    "_source": {
        "48": "136587",
        "10018": 0,
        "10070": 0,
        "10071": 0,
        "10190": 0,
        "10191": 0,
        "path": "/home/elk/logstash-5.6.3/request",
        "@timestamp": "2017-11-22T06:55:51.335Z",
        "@version": "1",
        "host": "Ubuntu-20170424",
        "message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",
    "type": "136587"
    }
}

可以看到,原始记录被保存,同时字段也被解析保存。如果确认不需要保存原始记录内容,可以加设置:remove_field => [“message”]

其中特别需要注意解析json数据的内容,logstash会在向es插入数据时默认会在_source下增加type,host,path三个字段,如果json内容中本身也含有type,host,path字段,那么解析后将覆盖掉logstash默认的这三个字段,尤其是type字段,这个同时也是做为index/type用的,覆盖掉后,插入进es中的index/type就是json数据记录中的内容,将不再是logstash config中配置的type值。


本文由【waitig】发表在等英博客
本文固定链接:Logstash收集json格式日志文件如何写配置文件
欢迎关注本站官方公众号,每日都有干货分享!
等英博客官方公众号
点赞 (0)分享 (0)