• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号

fluent/fluentd-kubernetes-daemonset: Fluentd daemonset for Kubernetes and it Doc ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

fluent/fluentd-kubernetes-daemonset

开源软件地址(OpenSource Url):

https://github.com/fluent/fluentd-kubernetes-daemonset

开源编程语言(OpenSource Language):

Ruby 62.5%

开源软件介绍(OpenSource Introduction):

Fluentd Daemonset for Kubernetes

README.md is generated from templates/README.md.erb

Docker Stars Docker Pulls ImageLayers Size ImageLayers Layers

Supported tags and respective Dockerfile links

See also dockerhub tags page: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/tags

Debian

Current stable

As the current limitation about the number of automated builds on hub.docker.com, there are some restrictions to ship daemonset images:

  • stackdriver, papertrail, syslog images (x86_64/arm64) won't be published anymore
  • logentries, loggly, logzio, s3 arm64 images won't be published anymore (x86_64 only supported)

If you want to use above non published images, build it by yourself. Dockefile itself is still maintained in this repository.

Multi-Arch images
  • Azureblob
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-azureblob-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-azureblob-1
  • Elasticsearch7
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-elasticsearch7-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-elasticsearch7-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
  • Opensearch
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-opensearch-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-opensearch-1
  • Cloudwatch
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-cloudwatch-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-cloudwatch-1
  • Forward
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-forward-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-forward-1
  • Gcs
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-gcs-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-gcs-1
  • Graylog
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-graylog-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-graylog-1
  • Kafka
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kafka-1.2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kafka-1
  • Kafka2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kafka2-1.2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kafka2-1
  • Kinesis
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kinesis-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kinesis-1
x86_64 images
  • Azureblob Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-azureblob-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-azureblob-amd64-1
  • Elasticsearch7 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-elasticsearch7-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-elasticsearch7-amd64-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-amd64
  • Elasticsearch6 Dockerfile
  • Opensearch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-opensearch-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-opensearch-amd64-1
  • Loggly Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-loggly-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-loggly-amd64-1
  • Logentries Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-logentries-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-logentries-amd64-1
  • Cloudwatch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-cloudwatch-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-cloudwatch-amd64-1
  • Stackdriver Dockerfile
  • S3 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-s3-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-s3-amd64-1
  • Syslog Dockerfile
  • Forward Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-forward-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-forward-amd64-1
  • Gcs Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-gcs-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-gcs-amd64-1
  • Graylog Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-graylog-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-graylog-amd64-1
  • Papertrail Dockerfile
  • Logzio Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-logzio-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-logzio-amd64-1
  • Kafka Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kafka-amd64-1.2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kafka-amd64-1
  • Kafka2 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kafka2-amd64-1.2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kafka2-amd64-1
  • Kinesis Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kinesis-amd64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kinesis-amd64-1
arm64 images
  • Azureblob Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-azureblob-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-azureblob-arm64-1
  • Elasticsearch7 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-elasticsearch7-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-elasticsearch7-arm64-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-arm64
  • Elasticsearch6 Dockerfile
  • Opensearch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-opensearch-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-opensearch-arm64-1
  • Loggly Dockerfile
  • Logentries Dockerfile
  • Cloudwatch Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-cloudwatch-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-cloudwatch-arm64-1
  • Stackdriver Dockerfile
  • S3 Dockerfile
  • Syslog Dockerfile
  • Forward Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-forward-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-forward-arm64-1
  • Gcs Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-gcs-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-gcs-arm64-1
  • Graylog Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-graylog-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-graylog-arm64-1
  • Papertrail Dockerfile
  • Logzio Dockerfile
  • Kafka Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kafka-arm64-1.2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kafka-arm64-1
  • Kafka2 Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kafka2-arm64-1.2
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kafka2-arm64-1
  • Kinesis Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15.1-debian-kinesis-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.15-debian-kinesis-arm64-1

You can also use v1-debian-PLUGIN tag to refer latest v1 image, e.g. v1-debian-elasticsearch. On production, strict tag is better to avoid unexpected update.

See dockerhub's tags page for older tags.

Old stable

v0.12 development has been ended. These images are never updated.

Alpine Linux (This is deprecated. Use Debian images instead)

What is Fluentd?

Fluentd Logo

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

www.fluentd.org

Image versions

Fluentd versioning is as follows:

Series Description
v1.x current stable
v0.12 Old stable, no longer updated

Settings

Default image version

Default YAML uses latest v1 images like fluent/fluentd-kubernetes-daemonset:v1-debian-kafka. If you want to avoid unexpected image update, specify exact version for image like fluent/fluentd-kubernetes-daemonset:v1.8.0-debian-kafka-1.0.

Run as root

This is for v0.12 images.

In Kubernetes and default setting, fluentd needs root permission to read logs in /var/log and write pos_file to /var/log. To avoid permission error, you need to set FLUENT_UID environment variable to 0 in your Kubernetes configuration.

Use your configuration

These images have default configuration and support some environment variables for parameters but it sometimes doesn't fit your case. If you want to use your configuration, use ConfigMap feature.

Each image has following configurations:

  • fluent.conf: Destination setting, Elaticsearch, kafka and etc.
  • kubernetes.conf: k8s specific setting. tail input for log files and kubernetes_metadata filter
  • tail_container_parse.conf: parser setting for /var/log/containers/*.log. See also "Use CRI parser for containerd/cri-o" logs section
  • prometheus.conf: prometheus plugin for fluentd monitoring
  • systemd.conf: systemd plugin for collecting systemd-journal log. See also "Disable systemd input" section.

Overwrite conf file via ConfigMap. See also several examples:

Use CRI parser for containerd/cri-o logs

This feature is available since v1.12.0-xxx-1.1.

By default, these images use json parser for /var/log/containers/ files because docker generates json formatted logs. On the other hand, containerd/cri-o use different log format. To parse such logs, you need to use cri parser instead.

You can use cri parser by overwriting tail_container_parse.conf via ConfigMap.

# configuration example
<parse>
  @type cri
</parse>

See also CRI parser README

Use FLUENT_CONTAINER_TAIL_PATH to change container logs folder

You can update the default path for the container logs i.e /var/log/container/*.log and also one can add multiple path as defined in this fluentd document https://docs.fluentd.org/input/tail#path

Use FLUENT_CONTAINER_TAIL_EXCLUDE_PATH to exclude specific container logs

Since v1.9.3 or later images.

You can exclude container logs from /var/log/containers/ with FLUENT_CONTAINER_TAIL_EXCLUDE_PATH. If you have a trouble with specific log, use this envvar, e.g. ["/var/log/containers/logname-*"].

Disable systemd input

If you don't setup systemd in the container, fluentd shows following messages by default configuration.

[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s

You can suppress these messages by setting disable to FLUENTD_SYSTEMD_CONF environment variable in your kubernetes configuration.

Disable prometheus input plugins

By default, latest images launch prometheus plugins to monitor fluentd. You can disable prometheus input plugin by setting disable to FLUENTD_PROMETHEUS_CONF environment variable in your kubernetes configuration.

Disable sed execution on elasticsearch image

This is for older images. Latest elasticsearch images don't use sed.

By historical reason, elasaticsearch image executes sed command during startup phase when FLUENT_ELASTICSEARCH_USER or FLUENT_ELASTICSEARCH_PASSWORD is specified. This sometimes causes a problem with read only mount. To avoid this problem, set "true" to FLUENT_ELASTICSEARCH_SED_DISABLE environment variable in your kubernetes configuration.

Running on OpenShift

This daemonset setting mounts /var/log as service account fluentd so you need to run containers as privileged container. Here is command example:

oc project kube-system
oc create -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml
oc adm policy add-scc-to-user privileged -z fluentd
oc patch ds fluentd -p "spec:
  template:
    spec:
      containers:
      - name: fluentd
        securityContext:
          privileged: true"
oc delete pod -l k8s-app=fluentd-logging

This is from nekop's japanese article.

Note

kafka image doesn't support zookeeper parameters

zookeeper gem doesn't work on Debian 10, so kafka image doesn't include zookeeper gem.

Windows k8s daemonset not supported in this repository

Maintainers don't have k8s experience on Windows. Some users create k8s daemonset on Windows:

Please check them out.

kafka image suggestion

Using debian-kafka2/debian-kafka2-arm64 images are better than using debian-kafka/debian-kafka-arm64 images. Because debian-kafka2/debian-kafka2-arm64 images use out_kafka2 plugin but debian-kafka/debian-kafka-arm64 images use deprecated out_kafka_buffered plugin.

Maintainers

Some images are contributed by users. If you have a problem/question for following images, ask it to contributors.

  • azureblob : @elsesiy
  • papertrail : @alexouzounis
  • kafka : @erhudy
  • graylog : @rtnpro
  • gcs : @andor-pierdelacabeza
  • Amazon Kinesis : @shiftky
  • logz.io : @SaMnCo / @jamielennox
  • splunkhec: @FutureSharks

Currently, we don't accept new destination request without contribution. See #293

References

Kubernetes Logging with Fluentd

Issues

We can't notice comments in the DockerHub so don't use them for reporting issues or asking question.

If you have any problems with or questions about this image, please contact us through a GitHub issue.

Pull Request

Update templates files instead of docker-image files. docker-image files are automatically generated from templates.

Note: This file is generated from templates/README.md.erb




鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
LevelUpEducation/kubernetes-demo发布时间:2022-08-13
下一篇:
nicolaka/netshoot: a Docker + Kubernetes network trouble-shooting swiss-army con ...发布时间:2022-08-13
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap