Enabling Dual-stack on Existing IPv4-only Cluster

[optix2000]

I have a very old k3s cluster that started on 1.21 and is now on 1.30 that I would like to upgrade to using dual-stack.

Docs state that: Dual-stack networking must be configured when the cluster is first created. It cannot be enabled on an existing cluster once it has been started as IPv4-only.

However, these docs are 2 years old (at time of writing), and there doesn't seem to be any indication why this wouldn't be possible, given that regular k8s lets you do an in place "upgrade" to dual-stack.

Is there any technical reason why this is not possible?

Ignoring the docs and just blindly setting cluster-cidr with IPv6 addresses doesn't work:

--cluster-cidr=10.42.0.0/16,fd00:aaaa::/64
--service-cidr=10.43.0.0/16,fd00:aaaa:0:1::/112

K3's knows that nodes have an IPv6 address (as the annotations show IPv6) k3s.io/internal-ip: 38.46.x.x,2602:x:x:x::1:0
But k3s doesn't use the IPv6 addresses, even though the underlying nodes have IPv6 addresses and have IPv6 connectivity.

Addresses:
  InternalIP:  38.46.x.x
  Hostname:    foo.bar.baz
...
PodCIDR:                      10.42.1.0/24
PodCIDRs:                     10.42.1.0/24

tldr: What stops us from enable Dual-Stack after cluster creation?

[brandond]

Nodes are only assigned Pod CIDRs when they first join the cluster. There is no logic in the Kubernetes IPAM controller to assign a new IPv6 CIDR if the cluster is later reconfigured to enable IPv6 support.

You can try deleting and rejoining the nodes, so that they get a new CIDR assigned, but this is not something we test or support.

[optix2000]

Thanks for the response!

Does this mean that a CNI that doesn't respect podCIDR could also work around this restriction if all the pods get recreated?

[immanuelfodor]

@optix2000 have you tried it? What are your results?

[optix2000]

I hacked it in by modifying the IP ranges in kine/etcd so I wouldn't have to recreate the nodes. Then I just recreated the pods with ipv6 enabled. Everything has been working without a hitch, though I'm using Cilium and not the built in CNI.

[immanuelfodor]

@optix2000 That's promising! Thank you! Can you explain modifying the IP ranges in kine/etcd in more detail? What/How did you edit it?

[optix2000]

Pretty much you can connect to etcd that k3s connects to and modify the CIDR's of the node in the registry directly, bypassing the k8s API. On restart k3s will load the (now changed) node configs. I don't remember the exact keys, but the structure is very logical and you can quickly find the right keys with a quick table scan, or read the k8s code to see what key it writes to.

If you're not familiar with the internals of k8s I would highly recommend against it. Screwing something up here will brick your k8s cluster.

[long2ice]

So how can do this?

[brandond]

As I said above:

you can try deleting and rejoining the nodes, so that they get a new CIDR assigned, but this is not something we test or support.

[immanuelfodor]

@long2ice have you tried it? What are your results?