elasticsearch - Filebeat无法加载输入(filebeat can't load input)

Question

filebeat loading input is 0 and filebeat don't have any log.

(filebeat加载输入为0，且filebeat没有任何日志。)

filebeat should read inputs that are some logs and send it to logstash.

(filebeat应该读取一些日志输入，并将其发送到logstash。)

i have some filters in logstash.conf, but i removed it temporarily.

(我在logstash.conf中有一些过滤器，但我暂时将其删除。)

and logstash send these to elastic and finally kibana.

(然后用logstash将它们发送到弹性的，最后是木版。)

filebeat.config.modules: 
  path: "${path.config}/modules.d/*.yml"
  reload.enabled: true
  reload.period: 10s
filebeat.inputs: 
  enabled: true
  paths: 
    - /var/log/TestLog/*.log
  type: log
filebeat.registry.path: /var/lib/filebeat/registry/filebeat
logging.files: 
  name: filebeat.log
  path: /var/log/filebeat
logging.level: info
logging.selectors: 
  - "*"
logging.to_files: true
monitoring.enabled: false
output.logstash: 
  enabled: true
  hosts: 
    - "192.168.80.20:5044"
setup.kibana: ~
setup.template.settings: 
  index.number_of_shards: 1

  ask by samen translate from so

Answer 1

response of journalctl -fu filebeat is

(journalctl -fu filebeat的响应是)

INFO        instance/beat.go:422        filebeat start running.
INFO        registrar/migrate.go:104        No registry home found. Create: /var/lib/filebeat/registry/filebeat/filebeat
INFO        registrar/migrate.go:112        Initialize registry meta file
INFO        registrar/registrar.go:108        No registry file found under: /var/lib/filebeat/registry/filebeat/filebeat/data.json. Creating a new registry file.
INFO        registrar/registrar.go:145        Loading registrar data from /var/lib/filebeat/registry/filebeat/filebeat/data.json
INFO        registrar/registrar.go:152        States Loaded from registrar: 0
WARN        beater/filebeat.go:368        Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
INFO        crawler/crawler.go:72        Loading Inputs: 0
INFO        crawler/crawler.go:106        Loading and starting Inputs completed. Enabled inputs: 0
INFO        cfgfile/reload.go:171        Config reloader started
INFO        log/input.go:148        Configured paths: [/var/log/elasticsearch/*_access.log /var/log/elasticsearch/*_audit.log /var/log/elasticsearch/*_audit.json]
INFO        log/input.go:148        Configured paths: [/var/log/elasticsearch/*_deprecation.log /var/log/elasticsearch/*_deprecation.json]
INFO        log/input.go:148        Configured paths: [/var/log/elasticsearch/gc.log.[0-9]* /var/log/elasticsearch/gc.log]
INFO        log/input.go:148        Configured paths: [/var/log/elasticsearch/*.log /var/log/elasticsearch/*_server.json]
INFO        log/input.go:148        Configured paths: [/var/log/elasticsearch/*_index_search_slowlog.log /var/log/elasticsearch/*_index_indexing_slowlog.log /var/log/elasticsearch/*_index_search_slowlog.json /var/log/elasticsearch/*_index_indexing_slowlog.json]
INFO        input/input.go:114        Starting input of type: log; ID: 10720371839583549447
INFO        input/input.go:114        Starting input of type: log; ID: 8161597721645621668
INFO        input/input.go:114        Starting input of type: log; ID: 15537576637552474368
INFO        input/input.go:114        Starting input of type: log; ID: 14070679154152675563
INFO        input/input.go:114        Starting input of type: log; ID: 7953850694515857477
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_audit.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_audit.json
INFO        log/input.go:148        Configured paths: [/var/log/logstash/logstash-plain*.log]
INFO        log/input.go:148        Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]
INFO        input/input.go:114        Starting input of type: log; ID: 17306378383715639109
INFO        input/input.go:114        Starting input of type: log; ID: 14725834876846155099
INFO        log/harvester.go:253        Harvester started for file: /var/log/logstash/logstash-plain.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_index_search_slowlog.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/logstash/logstash-slowlog-plain.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_deprecation.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_deprecation.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.27
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_server.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_deprecation.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.31
INFO        log/input.go:148        Configured paths: [/var/log/auth.log* /var/log/secure*]
INFO        log/input.go:148        Configured paths: [/var/log/messages* /var/log/syslog*]
INFO        input/input.go:114        Starting input of type: log; ID: 14797590234914819083
INFO        input/input.go:114        Starting input of type: log; ID: 16974178264304869863
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_deprecation.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_server.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/secure-20191201
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/messages-20191117
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_index_indexing_slowlog.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.02
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/messages-20191124
INFO        log/harvester.go:253        Harvester started for file: /var/log/secure
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_index_search_slowlog.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.03
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.08
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.18
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.11
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.26
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.06
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.12
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.20
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.29
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.21
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.07
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.13
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.19
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.28
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.22
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.24
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.23
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.05
INFO        log/harvester.go:253        Harvester started for file: /var/log/secure-20191110
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.09
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.10
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_index_search_slowlog.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.14
INFO        log/harvester.go:253        Harvester started for file: /var/log/secure-20191117
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.16
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.30
INFO        log/harvester.go:253        Harvester started for file: /var/log/secure-20191124
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.01
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.04
INFO        log/harvester.go:253        Harvester started for file: /var/log/messages-20191201
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.15
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.17
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.00
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log.25
INFO        log/harvester.go:253        Harvester started for file: /var/log/messages
INFO        log/harvester.go:253        Harvester started for file: /var/log/messages-20191110
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/elasticsearch_index_search_slowlog.json
INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/my-application_index_indexing_slowlog.log
INFO        pipeline/output.go:95        Connecting to backoff(async(tcp://192.168.80.20:5044))
INFO        pipeline/output.go:105        Connection to backoff(async(tcp://192.168.80.20:5044)) established
INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":550,"time":{"ms":560}},"total":{"ticks":4600,"time":{"ms":4612},"value":4600},"user":{"ticks":4050,"time":{"ms":4052}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":70},"info":{"ephemeral_id":"e901ac2b-21fa-47b1-a84d-3ddc10b068fd","uptime":{"ms":30285}},"memstats":{"gc_next":57786240,"memory_alloc":50264424,"memory_total":511186464,"rss":92864512},"runtime":{"goroutines":387}},"filebeat":{"events":{"active":4139,"added":34923,"done":30784},"harvester":{"open_files":64,"running":64,"started":64}},"libbeat":{"config":{"module":{"running":0},"reloads":2},"output":{"events":{"acked":30720,"active":4096,"batches":17,"total":34816},"read":{"bytes":96},"type":"logstash","write":{"bytes":5233807}},"pipeline":{"clients":9,"events":{"active":4119,"filtered":64,"published":34836,"retry":2048,"total":34903},"queue":{"acked":30720}}},"registrar":{"states":{"current":63,"update":30784},"writes":{"success":48,"total":48}},"system":{"cpu":{"cores":2},"load":{"1":3.55,"15":3.97,"5":4.77,"norm":{"1":1.775,"15":1.985,"5":2.385}}}}}}