spring boot - Nomad and consul setup

Question

Should I run consul slaves alongside nomad slaves or inside them? The later might not make sense at all but I'm asking it just in case.

I brought my own nomad cluster up with consul slaves running alongside nomad slaves (inside worker nodes), my deployable artifacts are docker containers (java spring applications). The issue with my current setup is that my applications can't access consul slaves (to read configurations) (none of 0.0.0.0, localhost, worker node ip worked)

Lets say my service exposes 8080, I configured docker part (in hcl file) to use bridge as network mode. Nomad maps 8080 to 43210. Everything is fine until my service tries to reach the consul slave to read configuration. Ideally giving nomad worker node IP as consul host to Spring should suffice. But for some reason it's not.

I'm using latest version of nomad.

I configured my nomad slaves like https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/nomad/client1.hcl

And the link below shows how I configured/ran my consul slave: https://github.com/bmd007/statefull-geofencing-faas/blob/master/infrastructure/server2.yml

Note: if I use static port mapping and host as the network mode for docker (in nomad) I'll be fine but then I can't deploy more than one instance of each application in each worker node (due to port conflic)

Answer 1

Nomad jobs listen on a specific host/port pair.

You might want to ssh into the server and run docker ps to see what host/port pair the job is listening on.

a93c5cb46a3e   image-name bash   2 hours ago    Up 2 hours  10.0.47.2:21435->8000/tcp, 10.0.47.2:21435->8000/udp foo-bar

Additionally, you will need to ensure that the consul nomad job is listening on port 0.0.0.0, or the specific ip of the machine. I believe that is this config value: https://www.consul.io/docs/agent/options.html#_bind

All those will need to match up in order to consul to be reachable.

More generally, I might recommend: if you're going to run consul with nomad, you might want to switch to host networking, so that you don't have to deal with the specifics of the networking within a container. Additionally, you could schedule consul as a system job so that it is automatically present on every host.