I'm attempting to containerize an application that depends on Kafka but I'm running into issues with my application attempting to establish a connection with Kafka on localhost:9092 once they are running in containers.
The stack has 2 containers that need to communicate with each other defined in the docker-compose file below. I'm creating a new network kafka-net using the bridge driver.
version: '3.0'
networks:
kafka-net:
driver: bridge
services:
spotify-kafka:
image: 'spotify/kafka'
container_name: 'spotify-kafka'
networks:
- kafka-net
ports:
- '2181:2181'
- '9092:9092'
environment:
- ADVERTISED_HOST=localhost
- ADVERTISED_PORT=9092
kafkamirror:
image: 'kafkamirror/kafkamirror'
container_name: 'kafkamirror'
networks:
- kafka-net
ports:
- '3000:3000'
- '8080:8080'
depends_on:
- spotify-kafka
With this configuration, I can spin up both containers, docker attach
into the CLI, and ping each container from one another just fine. A docker network inspect kafka-simulator_kafka-net
gives me the following:
[
{
"Name": "kafka-simulator_kafka-net",
"Id": "a53e3972edc6c67bf6cfa6fdd1fc2c48796e9da1e8cabf0710cb1270eb53a1fe",
"Created": "2021-01-07T02:42:32.744642052Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.29.0.0/16",
"Gateway": "172.29.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"10858507278d74479ca49b9f81c69f1ed71abbaf4915b17168cbce7797427153": {
"Name": "spotify-kafka",
"EndpointID": "acb8b94a73205a774aa15a491523ab91f40f5b7fc9e281dfebf6aaa024748035",
"MacAddress": "02:42:ac:1d:00:02",
"IPv4Address": "172.29.0.2/16",
"IPv6Address": ""
},
"c478714194b4d4fea144e277c92952ff576feb37f16dbf831b0da148233cf371": {
"Name": "kafkamirror",
"EndpointID": "fe2be0eae705b4728c4153bc07e8d00f330dc7d24931590754416888c4efad26",
"MacAddress": "02:42:ac:1d:00:03",
"IPv4Address": "172.29.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "kafka-net",
"com.docker.compose.project": "kafka-simulator",
"com.docker.compose.version": "1.27.4"
}
}
]
The containers appear to be on the same network and can communicate with one another, however in my application code, I'm attempting to create a connection to Kafka using localhost:9092 which doesn't work once they're containerized, since they have different IP addresses. I've tried using the "host" driver with no luck.
Any ideas on how to establish a connection with Kafka once it's containerized? Is there a way to statically set the IP addresses or somehow retrieve the allocated IP address on startup with a docker command and set that within my application code? I'm just not sure what IP address to use within the code of my application.