Today I learned that calico can easily get confused about its primary interface when adding a VPN interface to the host. Calico automatically detects the primary interface of a host and uses this interface to build its overlay network using IPIP, VXLAN or wireguard. This can result in unexpected behaviour like setting up double tunnelling through your VPN instead of using the local, physical network between your hosts, which has severe impacts on performance due to latency, bandwidth limitations and MTU size. The solution to the problem was as simple as removing the VPN from all nodes and further drain and reboot all nodes. While the latter might be optional, it made sure, everything is clear.

felix/vxlan_mgr.go 565: VXLAN device MTU needs to be updated device="vxlan.calico" new=1230 old=1450

I came across this, because I became suspicious about log messages by calico talking about setting a MTU of 1230 on its VXLAN interfaces, while debugging some internal networking issues. This was apparently the result of my recently added VPN that was supposed to allow simple access from a cloud server. In retro perspective it might obvious that this had to cause issues. Interestingly enough it only turned out to be problematic, once I deployed host firewall rules.