Clarrification on setting up embedded HA etcd

[ggeldenhuis]

I have been following https://docs.k3s.io/datastore/ha-embedded to setup a multi node k3s cluster.

The documentation says you should specify the server

--server https://<ip or hostname of server1>:6443 \

when setting up the second and subsequent master nodes.

If you configure the first node with --tls-san value however, and you specify the actual server value then the setup does not work, and you get TLS related issues with the server name being missing from the certificate. This happens on the second server that you are trying to join to the etcd cluster.

So was wondering if it would be useful to expand the documentation a little bit to specify the different behaviour when using --tls-san?

Config that has worked for me:
The first server:

# k3s-c02-master01.example.com
curl -sfL https://get.k3s.io | sh -s - server \
  --token=secrettext \
  --cluster-init \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san=k3s-c02-vip.example.com 

The second and third server:

# k3s-c02-master02.example.com
curl -sfL https://get.k3s.io | sh -s - server \
  --token=secrettext \
  --server https://k3s-c02-vip.example.com:6443 \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san=k3s-c02-vip.example.com 

The workload node:

curl -sfL https://get.k3s.io | K3S_TOKEN=secrettext sh -s - agent --server https://k3s-c02-vip.example.com:6443

The VIP address is HAproxy load balancing across the 3 master servers.

This was tested with k3s v1.34.5+k3s1 and on Ubuntu 22.04.

Somewhat related question, the k3s docs does not actually say how to interact with the etcd cluster. I have been using the tools installed by apt install etcd-client. I read #9841 which suggested that:
"While you can interact directly with the embedded etcd, we recommend against it." but does not actually specify what the alternative would be. Specifically I want to see which node is the leader and if the etcd cluster is healthy. I could not find an appropriate verb for k3s command that provides similar level of information to etcdctl endpoint status --cluster --write-out=table. The only available verb is etcd-snapshot The above command did require me to set:

  export ETCDCTL_API=3
  export ETCDCTL_ENDPOINTS="https://127.0.0.1:2379"
  export ETCDCTL_CERT="/var/lib/rancher/k3s/server/tls/etcd/server-client.crt"
  export ETCDCTL_KEY="/var/lib/rancher/k3s/server/tls/etcd/server-client.key"

[brandond]

If you configure the first node with --tls-san value however, and you specify the actual server value then the setup does not work, and you get TLS related issues with the server name being missing from the certificate. This happens on the second server that you are trying to join to the etcd cluster.

What?

Hostnames and IP addresses for cluster members are automatically added to the SAN list for the certificate that is served on port 9345. You only need to manually add entries via --tls-san if you want to access the host via a hostname or IP address not associated with a cluster member - such as a DNS alias or load-balanced VIP.

You DO NOT need to add --tls-san entries for ANY cluster members; secondary or otherwise.

It sounds like you're trying to set the --server URL to an address for the node that doesn't match the hostname autodetected by Kubernetes - did you want to override it with --node-name?

[ggeldenhuis]

I double checked, the documentation says you should use the --server https://<ip or hostname of server1>:6443 but that fails when you run it on server2:

Error message:

Mar 23 08:42:57 k3s-c02-master02 k3s[293080]: time="2026-03-23T08:42:57Z" level=fatal msg="Error: preparing server: failed to bootstrap cluster data: failed to check if bootstrap data has been initialized: failed to validate token: CA cert validation failed: Get \"https://k3s-c02-master01.example.com:6443/cacerts\": tls: failed to verify certificate: x509: certificate is valid for k3s-c02-master01, k3s-c02-vip.example.com, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, localhost, not k3s-c02-master01.example.com"

Specifically it fails because I specified the tls-san value on server1. If you use the tls-san value in the --server value when building server2 then it all works as expected. So the documentation is at best unspecific and at worst wrong.

So working config looks like:

# k3s-c02-master01.example.com
curl -sfL https://get.k3s.io | sh -s - server \
  --token=secrettext \
  --cluster-init \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san=k3s-c02-vip.example.com 

# k3s-c02-master02.example.com
curl -sfL https://get.k3s.io | sh -s - server \
  --token=secrettext \
  --server https://k3s-c02-vip.example.com:6443 \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san=k3s-c02-vip.example.com 

non-working:

# k3s-c02-master01.example.com
curl -sfL https://get.k3s.io | sh -s - server \
  --token=secrettext \
  --cluster-init \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san=k3s-c02-vip.example.com 

# k3s-c02-master02.example.com
curl -sfL https://get.k3s.io | sh -s - server \
  --token=secrettext \
  --server https://k3s-c02-master01.example.com:6443 \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san=k3s-c02-vip.example.com 

[brandond]

Get "https://k3s-c02-master01.example.com:6443/cacerts\": tls: failed to verify certificate
certificate is valid for k3s-c02-master01, k3s-c02-vip.example.com

Yes, like I said:

you're trying to set the --server URL to an address for the node that doesn't match the hostname autodetected by Kubernetes - did you want to override it with --node-name?

If you want the certificate to be valid for the FQDN, you need to set the node's hostname and/or nodename to the FQDN. You can also add the FQDN as a TLS SAN but if you do that you're going to have to do the same for all the nodes in the cluster - you're better off just properly configuring the hostname on your nodes so that the correct entries are present in the cert by default.

[ggeldenhuis]

I am not sure we are on the same page... my working example is functioning perfectly, well at least as far as I am concerned because I specify the VIP address rather than an individual server which is more robust as far as I understand. The original reason for this post was that I personally found the documentation to be confusing/wrong. Maybe I still don't understand the matter.

[brandond]

I'm not sure what's confusing. All node hostnames and IPs are automatically added to the cert. If you set the hostname to a short name, then a short name is what goes on the cert. If you then try to address the host using the FQDN, the certificate will not be valid for the FQDN.

We do not want to tell people to add a bunch of extra stuff via --tls-san, when the correct fix is to set the hostname to match however you want to access the host.

[ggeldenhuis]

What is confusing is that if you follow the documentation to the letter you will not end up with a working cluster. Specifically https://docs.k3s.io/datastore/ha-embedded

My only goal is to help clarify the documentation so that me or anyone else like me don't make the same mistake.

Regarding hostnames, the hostname should only be the name of the machine there is difference between hostname and fqdn, they are not the same thing. machine01.example.com is a fqdn, machine01 is the hostname. If the k3s setup does not include the fqdn value then it either need to do so, or it needs to be documented that is does not and that you should specify it. The hostname command has a -f flag to distinguish between the two. hostnamectl only returns machine01 per this example. Red Hat used to argue the opposite, but I have not used it for many years, so not sure if that is still the case.

Ok, so you sort of answered me regarding tls-san. I am doing it correctly then, as in that I am specifying the VIP endpoint on server 1 and then using that in server2, etc. However it is not clear from the documentation.

I am happy to write a PR for documentation enhancement but wanted to clarify that I understand the behaviour.
Suggested changes:

• Be specify about the scenario when using a VIP
• Add a line that you need to ensure the shortname/hostname and fqdn is both added if you are not using a vip.

[brandond]

Regarding hostnames, the hostname should only be the name of the machine there is difference between hostname and fqdn, they are not the same thing. machine01.example.com is a fqdn, machine01 is the hostname

That's not correct. The hostname is whatever is stored by the kernel at /proc/sys/kernel/hostname. This MAY be a short name, or it MAY be a FQDN. That's up to you as the system administrator to configure.

For best results, you should set the hostname to the same value (short or fqdn) that you will use to refer to the host from other nodes.

[ggeldenhuis]

This ambiguity has frustrated me for big part of my career. Red Hat used to recommend setting the hostname as the fqdn. My copy of Advanced Programming in the UNIX Environment states:
"Historically, BSD-derived systems provide the gethostname function to return only the name of the host."
There is also mention of the maximum length of hostname to be 64 characters which is relatively short and could be interpreted as "must not include the domain".

We use fully qualified names when specifying infrastructure because we have multiple internal domains and don't want to rely on search domains.

I don't believe anyone should be required to modify their hostname naming convention to make software work. In this case it is suggested that one would need to do so. Which brings me back to asking if it is worthwhile making documentation changes to be more explicit about different types of configuration, something which I am willing to assist with. If the documentation were explicit then it would have negated the need for this whole conversation, and the age old "what should the hostname be" argument.