Categories
Development

Can pods continue to function when there is a single master with taints removed

From what I’ve read about Kubernetes, if the master(s) die, the workers should still be able to function as normal (https://stackoverflow.com/a/39173007/281469), although no new scheduling will occur.

However, I’ve found this to not be the case when the master can also schedule worker pods. Take a 2-node cluster, where one node is a master and the other a worker, and the master has the taints removed:

diagram

If I shut down the master and docker exec into one of the containers on the worker I can see that:

nc -zv ip-of-pod 80

succeeds, but

nc -zv ip-of-service 80

fails half of the time. The Kubernetes version is v1.15.10, using iptables mode for kube-proxy.

I’m guessing that since the kube-proxy on the worker node can’t connect to the apiserver, it will not remove the master node from the iptables rules.

Questions:

  1. Is it expected behaviour that kube-proxy won’t stop routing to pods on master nodes, or is there something “broken”?
  2. Are any workarounds available for this kind of setup to allow the worker nodes to still function correctly?

I realise the best thing to do is separate the CP nodes but that’s not viable for what I’m working on at the moment.

Leave a Reply

Your email address will not be published. Required fields are marked *