olap - 如何提高Clickhouse Kafka消费者的性能(How to improve performance on Clickhouse Kafka consumer)

Question

I have a Kafka topic of 9 partitions and even when I use 9 consumers, and set the streaming interval to 1 second, I still don't see the machine's resources putting much effort, CPU is even lower than 10% (for 8 cores, 16GB AMI).

(我有一个9个分区的Kafka主题，即使我使用9个使用者，并且将流传输间隔设置为1秒，我仍然看不到机器的资源投入太多，CPU甚至低于10％（对于8核， 16GB AMI）。)

Any other properties I should attempt to modify?

(我应该尝试修改其他属性吗？)

thanks in advance.

(提前致谢。)

  ask by roee zi translate from so

Answer 1

I found out the source of the issue, the streaming poll timeout and the flush interval.

(我发现了问题的根源，流式轮询超时和刷新间隔。)

once I added the following to users.xml and restarted, the ingestion rate increased significantly

(将以下内容添加到users.xml并重新启动后，摄取率显着提高)

 <stream_poll_timeout_ms>5000</stream_poll_timeout_ms>
        <stream_flush_interval_ms>1500</stream_flush_interval_ms>
        <max_streams_to_max_threads_ratio>4</max_streams_to_max_threads_ratio>