https://www.jianshu.com/p/58036059ab8d

Docker提供了很多logging driver,默认情况下使用的json-file,它会把容器打到stdout/stderr的日志收集起来存到json文件中,docker logs所看到的日志就是来自于这些json文件。

当有多个docker host的时候你会希望能够把日志汇集起来,集中存放到一处,本文讲的是如何通过fluentd logging driver配合fluentd来达成这一目标。

目标:

将standalone容器打到stdout/stderror的日志收集起来
收集的日志根据容器名分开存储
日志文件根据每天滚动
第一步:配置Fluentd实例
首先是配置文件fluent.conf:

#   file: busybox/inst-2.yyyy-MM-dd.log
<match *.*.*>
  @type              file
  path               /fluentd/log/${tag[0]}/inst-${tag[1]}
  append             true
  <format>
    @type            single_value
    message_key      log
  </format>
  <buffer tag,time>
    @type             file
    timekey           1d
    timekey_wait      10m
    flush_mode        interval
    flush_interval    30s
  </buffer>
</match>

# 处理standalone容器的日志
# input 
#   tag: busybox
# output
#   file: busybox/busybox.yyyy-MM-dd.log
<match *>
  @type              file
  path               /fluentd/log/${tag}/${tag}
  append             true
  <format>
    @type            single_value
    message_key      log
  </format>
  <buffer tag,time>
    @type             file
    timekey           1d
    timekey_wait      10m
    flush_mode        interval
    flush_interval    30s
  </buffer>
</match>

新建一个目录比如/home/ubuntu/container-logs,并赋予权限chmod 777 /home/ubuntu/container-logs。

然后启动Fluentd实例,这里使用的Docker方式:

docker run -it \
  -d \
  -p 24224:24224 \
  -v /path/to/conf/fluent.conf:/fluentd/etc/fluent.conf \
  -v /home/ubuntu/container-logs:/fluentd/log
  fluent/fluentd:v1.3

第二步:指定容器的logging driver
在启动容器的时候执行使用fluentd作为logging driver,下面以standalone容器举例:

docker run -d \
  ...
  --log-driver=fluentd \
  --log-opt fluentd-address=<fluentdhost>:24224 \
  --log-opt mode=non-blocking \
  --log-opt tag={{.Name}} \
  <image>

注意上面的–log-opt tag={{.Name}}参数。

如果是docker compose / docker stack deploy部署,则在docker-compose.yaml中这样做 :

version: "3.7"
x-logging:
  &default-logging
  driver: fluentd
  options:
    fluentd-address: <fluentdhost>:24224
    fluentd-async-connect: 'true'
    mode: non-blocking
    max-buffer-size: 4m
    tag: "{{.Name}}"
services:
  busybox:
    image: busybox
    logging: *default-logging

第三步:观察日志
到/home/ubuntu/container-logs目录下能够看到类似这样的目录结构:

.
└── <container-name>
    └── <container-name>.20190123.log

参考文档
Configure logging drivers
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.docker.com%2Fconfig%2Fcontainers%2Flogging%2Fconfigure%2F
Customize log driver output
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.docker.com%2Fconfig%2Fcontainers%2Flogging%2Flog_tags%2F
Use Fluentd logging driver
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.docker.com%2Fconfig%2Fcontainers%2Flogging%2Ffluentd%2F
Docker CLI - run
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.docker.com%2Fengine%2Freference%2Fcommandline%2Frun%2F
Fluentd
Fluentd - out_file
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.fluentd.org%2Fv1.0%2Farticles%2Fout_file
Fluentd - formatter_single_value
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.fluentd.org%2Fv1.0%2Farticles%2Fformatter_single_value
Fluentd - buf_file
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.fluentd.org%2Fv1.0%2Farticles%2Fbuf_file
Fluentd - buffer
https://links.jianshu.com/go?to=https%3A%2F%2Fdocs.fluentd.org%2Fv1.0%2Farticles%2Fbuffer-section
转自:https://chanjarster.github.io/post/collect-docker-log-by-fluentd/

上面的示例,在实践中,并没有完全采用,不过采用日期形式记录容器日志的 fluentd 配置信息基本完全采用。我采用独立的docker-compose编辑文件处理fluentd的启动

version: '3'
services:
    fluentd:
      image: cheungpat/fluentd-s3
      container_name: "fluented-s3"
      restart: always
      environment:
        - 'FLUENTD_CONF=fluent.conf'
      volumes:
        - ./fluentd/conf/fluent.conf:/fluentd/etc/fluent.conf
        - ./fluentd/container-logs:/fluentd/log 
      ports:
        - "24224:24224"
      container_name: fluentd_frame2019_test

其他输出日志的容器采用独立的编排

  frontdemo1:
    network_mode: "host"
    image: registry.cn-beijing.aliyuncs.com/djm/frontdemo1:latest
    expose:
      - "80"
    volumes:  
      - ./front/wwwapp:/app/wwwapp
      - ./front/appsettings.json:/app/appsettings.json
    restart: always
    container_name: front_frame2019_test
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: front_frame2019_test
文档更新时间: 2020-09-22 02:02   作者:月影鹏鹏