Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
976 views
in Technique[技术] by (71.8m points)

olap - 如何提高Clickhouse Kafka消费者的性能(How to improve performance on Clickhouse Kafka consumer)

I have a Kafka topic of 9 partitions and even when I use 9 consumers, and set the streaming interval to 1 second, I still don't see the machine's resources putting much effort, CPU is even lower than 10% (for 8 cores, 16GB AMI).

(我有一个9个分区的Kafka主题,即使我使用9个使用者,并且将流传输间隔设置为1秒,我仍然看不到机器的资源投入太多,CPU甚至低于10%(对于8核, 16GB AMI)。)

Any other properties I should attempt to modify?

(我应该尝试修改其他属性吗?)

thanks in advance.

(提前致谢。)

  ask by roee zi translate from so

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

I found out the source of the issue, the streaming poll timeout and the flush interval.

(我发现了问题的根源,流式轮询超时和刷新间隔。)

once I added the following to users.xml and restarted, the ingestion rate increased significantly

(将以下内容添加到users.xml并重新启动后,摄取率显着提高)

 <stream_poll_timeout_ms>5000</stream_poll_timeout_ms>
        <stream_flush_interval_ms>1500</stream_flush_interval_ms>
        <max_streams_to_max_threads_ratio>4</max_streams_to_max_threads_ratio>

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...