docker 部署logstash

发表于

这是我参与11月更文挑战的第4天，活动详情查看：2021最后一次更文挑战

今天主要想介绍一下logstash 这个工具，真的很不错，最近在工作中使用到了，对于日志传输收集等都很友好，可以很方便的同步数据到elasticsearch 或者 kafka等工具均可以，今天主要介绍同步文件数据到elasticsearch中

1
复制代码docker pull logstash:6.4.0

km_log_pattern 文件：

1
perl复制代码STIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}:?%{SECOND},?%{MSECONDS}

logstash.conf 配置参数：

读取 文件数据 并写入 elasticsearch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
yaml复制代码input {

  file {

    path => ["/home/work/testVolume/test_map_log/*.log","/home/work/testVolume/test_map_log/*.log"]

    type => "test_map_new"

    start_position => "beginning"

  }

}

\


filter {

  grok {

    patterns_dir => ["/config-dir/cmap_log_pattern"]

    match => {

      "message" => [

          "\[%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{MSECONDS:mill_seconds}\]\[user_id:%{GREEDYDATA:user_id},mobile:%{GREEDYDATA:user_mobile},status:%{GREEDYDATA:user_status},real_name:%{GREEDYDATA:real_name},email:%{GREEDYDATA:user_email},city:%{GREEDYDATA:user_city},permission_info:%{GREEDYDATA:permission_info},b_stree_permission:%{GREEDYDATA:b_stree_permission},together_permission:%{GREEDYDATA:together_permission},is_admin:%{GREEDYDATA:is_admin}\]\[URL:%{GREEDYDATA:uri}\]\[params:%{GREEDYDATA:params_json_content}\]",

          "\[%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{MSECONDS:mill_seconds}\]\[user_id:%{GREEDYDATA:user_id},mobile:%{GREEDYDATA:mobile},platformCompany:%{GREEDYDATA:platformCompany},real_name:%{GREEDYDATA:real_name},email:%{GREEDYDATA:email},city:%{GREEDYDATA:city},role:%{GREEDYDATA:role},platformCompany:%{GREEDYDATA:platformCompany}\]\[URL:%{GREEDYDATA:uri}\]\[params:%{GREEDYDATA:params_json_content}\]",

          "\[%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{MSECONDS:mill_seconds}\]\[user_id:%{GREEDYDATA:user_id}\]\[URL:%{GREEDYDATA:uri}\]\[params:%{GREEDYDATA:params_json_content}\]"

      ]

    }

  }

  json {

    source => "params_json_content"

    target => "params_json"

    remove_field => ["paramsjson"]

  }

}

\


output {

  elasticsearch {

    hosts => ["127.0.0.1:9200"]

    index => "test_log"

    user => "test"

    password => "xxxxx"

  }

  stdout { codec => line }

}

读取kafka数据写入elasticsearch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ini复制代码input {
    kafka {
        bootstrap_servers => ["xxx.xxx.xxx.xxx:9092"]
        auto_offset_reset => "latest"
        consumer_threads => 5
        decorate_events => true
        group_id => "xxx"
        topics => ["xxxxxxxxxx"]
        type => "xxxxx"
    }
}

output {
    stdout {}
    elasticsearch {
          hosts => ["xxx.xxx.xxx.xxx:9200"]
          index => "kafka-xxx-%{+YYYY.MM.dd}"
    }
}

启动docker命令：

1
arduino复制代码docker run -d --name logstash_test  --log-opt max-size=10m --log-opt max-file=3  -v /config-dir:/config-dir -v /home/work/logstash_test/logstash:/home/work/logstash_test/logstash -v logstash -f /config-dir/logstash.conf

以上是通过读取文件然后写入elasticsearch 的方式去进行部署还有一种方式是通过部署logstash服务，其他服务进行服务调用去写入

相关logstash.conf 配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
ini复制代码input {

  tcp {

      host => "0.0.0.0"

      port => "5044"

      codec => json

  }

}

filter{

  if [type] == "logstash" {

        ruby { 

            code => "event.set('timestamp', event.timestamp.time.localtime.strftime('%Y-%m-%d %H:%M:%S'))" 

        }

    }

}

output {

  elasticsearch { 

             hosts => ["xx.xx.xx.xx:9200","xx.xx.xx.xx:9200"] #可以配置多个机器 一般为集群

             user => "xxxxxx" 

             password => "xxxxxx" 

             index => "xxxxxx" 

             codec => "json"

  }

  

  stdout { codec => json }

}

启动命令：

1
ruby复制代码docker run -it -d -p 5044:5044--name logstash --net somenetwork -v /docker/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml -v /docker/logstash/conf.d/:/usr/share/logstash/conf.d/ logstash:6.4.0

通过上面就可以搭建一个logstash的服务了，然后其他应用就可以直接调用 xx.xx.xx.xx:5044 传输日志文件进入elasticsearch 了

本文转载自: 掘金

开发者博客 – 和开发相关的 这里全都有

0%