java - Ignore logging of duplicated exception messages of third party libs

Question

I need to handle duplication of specific exeptions in my logs.

I use slf4j and logback for logging in my application. I use some external services(DB, apache kafka, third-party libs, etc.). When connection was lost to such service I got exception, such as

[kafka-producer-network-thread | producer-1] WARN  o.a.kafka.common.network.Selector - Error in I/O with localhost/127.0.0.1
java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_45]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_45]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:238) ~[kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [kafka-clients-0.8.2.0.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

The problem is that I got this message every second. This exception message will flood my log file, so I'll have a few GBs in log file in N hours.

I want to have log message about this exception once per 1-5 minutes. Is there any way to ignore duplication of exceptions in log file?

Possible solutions:

1. Ignore all logs for specific package and class. [bad, beucase I can skip important message]

2. Use http://logback.qos.ch/manual/filters.html#DuplicateMessageFilter [bad, because I can set only properties AllowedRepetitions or CacheSize. It'll match to all messages, but I need only specific exeption]

3. Write custom filter Maybe, you know already imlemented solutions?

Answer 1

I think your best option is simply extending the DuplicateMessageFilter you already found. It is not final and it is fairly easy to:

1. implement a new TurboFilter with a filter method based on classname or exceptionType or anything else you want to make your initial decision based upon

2. then delegate to the parent class for duplicity-check

The parameters you have available:

public FilterReply decide(Marker marker, Logger logger, Level level,
  String format, Object[] params, Throwable t) {

Include both Throwable and Logger.