From what I’ve read about Kubernetes, if the master(s) die, the workers should still be able to function as normal (https://stackoverflow.com/a/39173007/281469), although no new scheduling will occur.
However, I’ve found this to not be the case when the master can also schedule worker pods. Take a 2-node cluster, where one node is a master and the other a worker, and the master has the taints removed:
If I shut down the master and
docker exec into one of the containers on the worker I can see that:
nc -zv ip-of-pod 80
nc -zv ip-of-service 80
fails half of the time. The Kubernetes version is v1.15.10, using iptables mode for kube-proxy.
I’m guessing that since the kube-proxy on the worker node can’t connect to the apiserver, it will not remove the master node from the iptables rules.
- Is it expected behaviour that kube-proxy won’t stop routing to pods on master nodes, or is there something “broken”?
- Are any workarounds available for this kind of setup to allow the worker nodes to still function correctly?
I realise the best thing to do is separate the CP nodes but that’s not viable for what I’m working on at the moment.