minikube - DNS error with MySQL integration with kubernetes cluster

Question

What happened:
I am trying to create a service endpoint using the externalName spec to allow my microservices running inside the pods to access a local MySQL server on my local host.

This is the relevant section for the yaml file:

apiVersion: v1 
kind: Service 
metadata: 
  name: mysql 
  namespace: default 
spec: 
  type: ExternalName 
  externalName: host.minikube.internal

What you expected to happen:
I expect to be able to connect but my SpringBoot containers are showing that the mysql connection is not working. I have tested the microservices and it is working in Docker with the same MySQL database.

How to reproduce it (as minimally and precisely as possible):
Normal installation of minikube and kubernetes, running the dnsutils image from https://k8s.io/examples/admin/dns/dnsutils.yaml with the mysql service given above.

Anything else we need to know?:
I have tested out the troubleshooting detailed here (https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) but it did not resolve the problem. When running:

kubectl exec -i -t dnsutils -- nslookup mysql.default

I get the following message:

Server: 10.96.0.10
Address: 10.96.0.10#53

mysql.default.svc.cluster.local canonical name = host.minikube.internal.
** server can't find host.minikube.internal: SERVFAIL

command terminated with exit code 1

I have verified that CoreDNS is installed and running:

NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-z58cr 1/1 Running 0 31m

Endpoints are exposed:

NAME ENDPOINTS AGE
kube-dns 172.17.0.2:53,172.17.0.2:53,172.17.0.2:9153 32m

My /etc/resolv.conf only has one entry:

nameserver 192.168.53.145

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"windows/amd64"}

Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:09:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
Local Windows 10 Pro x64 running Kubernetes with Minikube OS (e.g: cat /etc/os-release):
NAME=Buildroot VERSION=2020.02.7 ID=buildroot VERSION_ID=2020.02.7 PRETTY_NAME="Buildroot 2020.02.7"

Kernel (e.g. uname -a): Linux minikube 4.19.150 #1 SMP Fri Nov 6 15:58:07 PST 2020 x86_64 GNU/Linux

Install tools: Installed using the relevant kubectl and minikube .exe files Network plugin and version (if this is a network-related bug):

Others:

Answer 1

This problem seems to be close related to Minikube issue described on github.

You can see, that in your Pod in the /etc/hosts file - there isn't any host.minikube.internal entry:

$ kubectl exec -it dnsutils -- cat /etc/hosts | grep "host.minikube.internal"
$

On the Minikube host you are able to reach host.minikube.internal because Minikube (version v1.10+) adds this hostname entry to /etc/hosts file. You can find more information in Host access | minikube.

This is example from my Minikube (I'm using docker driver):

user@minikube:~$ kubectl exec -it dnsutils -- cat /etc/hosts | grep "host.minikube.internal"
user@minikube:~$
user@minikube:~$ minikube ssh
docker@minikube:~$ cat /etc/hosts | grep host.minikube.internal 
192.168.49.1    host.minikube.internal
docker@minikube:~$ ping host.minikube.internal
PING host.minikube.internal (192.168.49.1) 56(84) bytes of data.
64 bytes from host.minikube.internal (192.168.49.1): icmp_seq=1 ttl=64 time=0.075 ms
64 bytes from host.minikube.internal (192.168.49.1): icmp_seq=2 ttl=64 time=0.067 ms

host.minikube.internal is only the entry in the /etc/hosts file, therefore nslookup can't correctly resolve it (nslookup queries name servers ONLY.).

docker@minikube:~$ nslookup host.minikube.internal
Server:         192.168.49.1
Address:        192.168.49.1#53

** server can't find host.minikube.internal: NXDOMAIN

The only workaround I think may help in some cases is adding hostAliases to Deployment/Pod manifest files:

...
spec:
  hostAliases:
  - ip: "192.168.49.1" # minikube IP
    hostnames:
    - "host.minikube.internal" # one or more hostnames that should resolve to the above address
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
...