Pod IP on VM node not accessible from other nodes

[drpump]

I'm trying to deploy the Garage S3-compatible service into my cluster using this operator: https://github.com/rajsinghtech/garage-operator (kudos to the dev, by the way). I've found that if a pod is scheduled on my VM node, the IP address is not route-able for the other nodes. Any suggestions for what could be wrong or how I can troubleshoot further would be most welcome.

Details

When deploying a 3-node garage cluster, the pod scheduled on my VM node gets an IP address that is not accessible from other nodes. It seems to be specific to garage or the pod configuration, because other pods scheduled on the VM node are accessible. I've worked around the issue by cordoning the VM node and redeploying: this gives me 3 working pods (2 x AMD, 1 x ARM).

I have 5 nodes in total:

• all Ubuntu 24.04
• 4 running directly on the hardware
• 1 running in a VMWare Fusion VM on a Mac M4 node
• the 4 hardware nodes are 2 ARM and 2 AMD
• k3s Is at v1.33.6+k3s1 on all nodes
• 3 server nodes using etcd (the VM node is a server node)
• using host-gw for networking
• the VM host is connected via wifi with bridged networking (other nodes are connected to an ethernet switch)

I deployed a couple of netshoot pods (network troubleshooting image) to troubleshoot. Briefly:

tracepath from netshoot on another node to the VM node garage pod:

# tracepath 10.42.3.9
 1?: [LOCALHOST]                      pmtu 1500
 1:  10.42.1.1                                             0.046ms 
 1:  10.42.1.1                                             0.031ms 
 2:  no reply
 3:  no reply

tracepath from netshoot on same node:

# tracepath 10.42.3.9
 1?: [LOCALHOST]                      pmtu 1500
 1:  garage-1.garage-headless.garage.svc.cluster.local     0.404ms reached
 1:  garage-1.garage-headless.garage.svc.cluster.local     0.077ms reached
     Resume: pmtu 1500 hops 1 back 1 

tracepath from netshoot on another node to CNPG (Postgres) pod on the VM node

tracepath 10.42.3.5
 1?: [LOCALHOST]                      pmtu 1500
 1:  10.42.1.1                                             0.051ms 
 1:  10.42.1.1                                             0.027ms 
 2:  gemini                                               14.924ms 
 3:  10-42-3-5.authentik-cnpg-r.auth.svc.cluster.local   151.353ms reached
     Resume: pmtu 1500 hops 3 back 3 

Other notes:

• I also checked the iptable rules on the VM node, and there is no difference between the CNPG pod and the garage pod rules.
• The garage cluster is deployed as a StatefulSet. CNPG doesn't use statefulsets, nor did the other pods I tested on the VM node. But I would not have expected this to make any difference to networking.
• The garage cluster uses the default service account, whereas cnpg has a custom configuration

[brandond]

Is the Garage operator deploying network policies that do not allow access to the pod from nodes?

[brandond]

Yeah that's not going to be it. There is some other issue with CNI traffic between your nodes.

VM host is on Wifi with the VM is using bridged networking so is directly visible on the same logical network as the other hosts. The other hosts are connected to the WiFi router using a ethernet switch, so not sure if that qualifies as the “same physical medium”.

This is unlikely to work properly. vxlan CAN work over wifi, but you are likely getting packet drops due to smaller MTU on the wireless network. Get all your nodes on the same network with the same MTU, or decrease the flannel vxlan MTU to the smallest size supported by any of your nodes.

You really can't just connect all your nodes to a wired network? This is never going to work GREAT for you.

[drpump]

This is unlikely to work properly … you are likely getting packet drops due to smaller MTU on the wireless network

While I understand that there are risks here, I’m consistently only having this problem with a specific pod despite node restarts and pod rescheduling. So this explanation seems unlikely unless that specific pod has networking setup that differs from the others.

Are you aware of anything in the pod configuration that might increase the risks? Would a wireguard-native network help here?

You really can't just connect all your nodes to a wired network?

The Mac Mini is a desktop machine with spare capacity and I don’t have any physical wiring nearby. I can perhaps connect it to a wifi mesh node with backhaul, but I assume this is likely to have the same issue. The Mac is a more powerful machine than the other nodes, so I’m reluctant to drop it out of the cluster. I could perhaps make it an agent node (and have been thinking about it), but I don’t think that would avoid this issue.

[drpump]

Checked the MTU settings: all nodes are using an MTU of 1500 for their native network interfaces. The Mac Mini host is also using MTU 1500 for its WiFi interface.

I also verified the interfaces being used for k3s networking. Turns out that one of the other nodes is also using wifi (when I set the node up it wasn’t connected to ethernet).

[drpump]

Poking around in routing tables and noticed that my VM node is assigning a static IP and has manually-specified base routes, which are different from the DHCP-provided routes on the other nodes. Will switch to DHCP with an IP reservation and see if this fixes the problem, but might take me a day or two to get to this.

[drpump]

I found a way to unblock, albeit quite manual:

1. Deploy a debug image inside the non-routable pod (kubectl debug -n garage garage2-1 -it --image=nicolaka/netshoot)
2. Run tcpdump -i eth0 imp in the debug container
3. Ping the pod from its host node to generate some traffic ping 10.42.3.50 (the pings will show up in the tcpdump output)

This unblocks the pod network interface and I am able to access it from my other nodes after these steps.

[drpump]

It seems that any outbound traffic from the pod unblocks the pod networking. So my slightly ugly fix is to run:

kubectl debug -n {namespace} {pod} -it --image=nicolaka/netshoot -- ping google.com -c 3

Note that the images that I've noticed having this issue (it appears to be quite image/deployment-specific):

• svclb-traefik
• garage
• longhorn engine-image-ei

[drpump]

Script to fix broken pods, run it from one of the other nodes:

#!/bin/sh
# Usage: `unblock.sh mynode`

kubectl get pods -A --field-selector spec.nodeName=$1 -o jsonpath='{range .items[*]}{.status.podIP} {.metadata.name} {.metadata.namespace}{"\n"}{end}' | while read -r line
do 
  ip=$(echo $line | cut -d" " -f1); 
  pod=$(echo $line | cut -d" " -f2)
  namespace=$(echo $line | cut -d" " -f3)
  ping ${ip} -c 1 -W 1 >/dev/null 2>&1
  if [ $? -eq 1 ]
  then
    echo "Pod ${pod} in namespace ${namespace} at address ${ip} is inaccessible, fixing ..."
    kubectl debug -n $namespace $pod -i --image=nicolaka/netshoot -- ping google.com -c 2
  else 
    echo "YAY! Pod ${pod} in namespace ${namespace} at address ${ip} is accessible"
  fi
done

[drpump]

Marking as an answer, but it's a workaround rather than an answer.

[brandond]

Really suggests that something is odd with your layer 2 networking. I've never seen any reports of anything like this. You might cross-post to the Flannel repo's issues, but I suspect this is not something that anyone else is going to be able to reproduce.

[drpump]

Thanks, yes it is quite weird. My guess is that it relates to the virtualization of the network device and is likely a race condition or at least timing-related. The layer 2 network, routes, iptables etc all check out correctly and there are no errors popping up in logs anywhere. Will ask a question on the flannel repo.