server cert compromised. tls: handshake message of length 70096 bytes exceeds maximum of 65536 bytes

[cheddarwhizzy]

Today one of my k3s cluster control planes started throwing tls: handshake message of length 70096 bytes exceeds maximum of 65536 bytes

Running echo | openssl s_client -showcerts -connect localhost:443 2>/dev/null | openssl x509 -text | grep DNS shows a ton of SANs that aren’t in the systemd file (--tls-san=127.0.0.1)

Is there a way to alter the SANs without access to the server?
Is there a way to reset these so they only contain the IPs and hostnames defined by the flag --tls-san?

See below for current SANs

54.188.29.57root@ip-172-30-241-154:/var/snap/amazon-ssm-agent/6312# echo | openssl s_client -showcerts -connect localhost:443  2>/dev/null | openssl x509 -text | grep DNS
                DNS:44.236.77.201:443, DNS:52.24.177.17:443, DNS:52.41.143.127:443, DNS:54.188.29.57:443, DNS:abcnews.tt.omtrdc.net, DNS:activisionblizzardin.tt.omtrdc.net, DNS:adobe.tt.omtrdc.net, DNS:aeroviasdelcontinent.tt.omtrdc.net, DNS:ally.tt.omtrdc.net, DNS:at.maxblue.de, DNS:audible.tt.omtrdc.net, DNS:autodesk.tt.omtrdc.net, DNS:axisbank.tt.omtrdc.net, DNS:bankofamerica.tt.omtrdc.net, DNS:barclaysbankplc.tt.omtrdc.net, DNS:bellca.tt.omtrdc.net, DNS:bestbuycanada.tt.omtrdc.net, DNS:bmwofnorthamericallc.tt.omtrdc.net, DNS:bootsuk.tt.omtrdc.net, DNS:canadianimperialbank.tt.omtrdc.net, DNS:capitalgroupcompanie.tt.omtrdc.net, DNS:capitaloneservices.tt.omtrdc.net, DNS:censusbureau.tt.omtrdc.net, DNS:ch32g61opmf7cq4pd8qg, DNS:ch32g6hopmf7b50k11ig, DNS:ch33uvpopmf7qtv8pj20, DNS:ch34mapopmf1no289sn0, DNS:ch3hn49opmfed9bjn840, DNS:ch3hn4popmfedlqe5em0, DNS:ch3scuhopmfa7v3h7vn0, DNS:ch3scupopmf8l7pcv0og, DNS:ch45s99opmf128suc2tg, DNS:ch47g99opmf20c2dcqpg, DNS:ch47ga1opmf2ffarrla0, DNS:ch48bn9opmfdvagcvj5g, DNS:ch49qipopmf8rgrh5330, DNS:ch4l2n9opmfda4lcm1jg, DNS:ch4l2n9opmfdch7l30l0, DNS:ch4qfv9opmf29dng1bhg, DNS:ch4qg01opmf2881dd1cg, DNS:ch4rd39opmffe01qe8g0, DNS:ch4rd3popmffb4g4hitg, DNS:ch5addhopmfdp3md8s00, DNS:ch5adepopmfe58bbcobg, DNS:ch5haqhopmf44j1oi6g0, DNS:ch5hbm1opmfel27on2p0, DNS:ch5n07hopmf4luqch6g0, DNS:ch5olgpopmf3iooopjl0, DNS:ch61vdhopmf2t2polma0, DNS:ch6l2jpopmf8ovndpe8g, DNS:ch6l2kpopmf97lsfkj60, DNS:ch6opshopmfb40tl1km0, DNS:ch6tv3popmf1odjthdeg, DNS:ch6v8jpopmfc9rgu1asg, DNS:ch7a4n1opmf12p798gc0, DNS:ch7a5ihopmf40aci0sbg, DNS:ch7b7opopmf2lobspjvg, DNS:ch7bs8popmf2pbt9s01g, DNS:ch7fd29opmf5dmq27970, DNS:ch7ofj9opmfcvgr2ut8g, DNS:ch7ogj9opmf0bfmbf5q0, DNS:ch7q591opmf01r0n6ihg, DNS:ch7ubi1opmf9m5q1eo4g, DNS:ch7ubihopmf9lek78k4g, DNS:ch807n1opmf610g14avg, DNS:ch8m3t9opmf6huuosmd0, DNS:ch8njd1opmf8drard5cg, DNS:ch8nkahopmf335965qq0, DNS:citiapac.tt.omtrdc.net, DNS:consumerinfo.tt.omtrdc.net, DNS:control.obscure-falls-6360.herokuspace.com, DNS:coopbank.tt.omtrdc.net, DNS:coty.tt.omtrdc.net, DNS:cvshealth.tt.omtrdc.net, DNS:dardenrestaurants.tt.omtrdc.net, DNS:dellinc.tt.omtrdc.net, DNS:deltaairlines.tt.omtrdc.net, DNS:discover.tt.omtrdc.net, DNS:dollargeneral.tt.omtrdc.net, DNS:dpcomdhl.tt.omtrdc.net, DNS:dpcomepost.tt.omtrdc.net, DNS:ec2-44-236-77-201.us-west-2.compute.amazonaws.com, DNS:ec2-52-24-177-17.us-west-2.compute.amazonaws.com, DNS:ec2-52-41-143-127.us-west-2.compute.amazonaws.com, DNS:ec2-54-188-29-57.us-west-2.compute.amazonaws.com, DNS:enterpriseholdingsin.tt.omtrdc.net, DNS:eonline.tt.omtrdc.net, DNS:equifax.tt.omtrdc.net, DNS:expressllc.tt.omtrdc.net, DNS:fifththirdbank.tt.omtrdc.net, DNS:generalmotorscorpora.tt.omtrdc.net, DNS:guitarcenter.tt.omtrdc.net, DNS:hallmarkcom.tt.omtrdc.net, DNS:harvardbusinessrevie.tt.omtrdc.net, DNS:homedepotcanada.tt.omtrdc.net, DNS:hpww.tt.omtrdc.net, DNS:hyattcorporation.tt.omtrdc.net, DNS:idfc.tt.omtrdc.net, DNS:ilotteryillinois.tt.omtrdc.net, DNS:incommholdingsgift.tt.omtrdc.net, DNS:indigoaviation.tt.omtrdc.net, DNS:invescogroup.tt.omtrdc.net, DNS:investorsbusinessdai.tt.omtrdc.net, DNS:k3s-college-adaptor-ci-lb-228526199.us-west-2.elb.amazonaws.com, DNS:k3s-college-adaptor.ci.ccctechcenter.org, DNS:kaiser.tt.omtrdc.net, DNS:kellogg.tt.omtrdc.net, DNS:kiamotorsamerica.tt.omtrdc.net, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:ldschurch.tt.omtrdc.net, DNS:lifetimefitness.tt.omtrdc.net, DNS:lloydsbankinggroup.tt.omtrdc.net, DNS:loblawsinc.tt.omtrdc.net, DNS:localhost, DNS:marvel.tt.omtrdc.net, DNS:master-0, DNS:master-1, DNS:master-ip-172-30-240-163, DNS:master-ip-172-30-241-154, DNS:mbox.wegmans.com, DNS:mboxedge.tt.omtrdc.net, DNS:mercedezbenzusallc.tt.omtrdc.net, DNS:metropcs.tt.omtrdc.net,DNS:microsoft.com, DNS:myfamilycominc.tt.omtrdc.net, DNS:netspendcorp.tt.omtrdc.net, DNS:nissanheliosna.tt.omtrdc.net, DNS:nvidia.tt.omtrdc.net, DNS:pepboys.tt.omtrdc.net, DNS:potterybarn.tt.omtrdc.net, DNS:pradaspa.tt.omtrdc.net, DNS:qantasairways.tt.omtrdc.net, DNS:registercom.tt.omtrdc.net, DNS:safewayinc.tt.omtrdc.net, DNS:salliemae.tt.omtrdc.net, DNS:saudicommissionforto.tt.omtrdc.net, DNS:scholasticinc.tt.omtrdc.net, DNS:scotiabank.tt.omtrdc.net, DNS:sjourney.aarp.org, DNS:snagajob.tt.omtrdc.net, DNS:soptimize.southwest.com, DNS:southwestairlines.tt.omtrdc.net, DNS:speed.cloudflare.com, DNS:starget.aircanada.com, DNS:starget.intel.com.tr, DNS:starget.intel.com.tw, DNS:starget.intel.fr, DNS:starget.intel.it, DNS:starget.panerabread.com, DNS:stt.dell.com, DNS:subway2016.tt.omtrdc.net, DNS:target.allianz.ch, DNS:target.bankofamerica.com, DNS:target.bose.com, DNS:target.champssports.com, DNS:target.footlocker.com, DNS:target.microsoft.com, DNS:target.thetruth.com, DNS:target.toyota.com, DNS:target.vwfs.de, DNS:target.walgreens.com, DNS:tdbankfinancialgroup.tt.omtrdc.net, DNS:three.tt.omtrdc.net, DNS:tiffanyandcompany.tt.omtrdc.net, DNS:transunion.tt.omtrdc.net, DNS:ttmetrics.jcpenney.com, DNS:twitterinc.tt.omtrdc.net, DNS:ups.tt.omtrdc.net, DNS:verizontelecom.tt.omtrdc.net, DNS:versacespa.tt.omtrdc.net, DNS:visibleservices.tt.omtrdc.net, DNS:walgreenco.tt.omtrdc.net, DNS:wgu.tt.omtrdc.net, DNS:wheelpros.tt.omtrdc.net, DNS:woolworthsfoodgroup.tt.omtrdc.net, DNS:workday.tt.omtrdc.net, DNS:wwgraingerinc.tt.omtrdc.net, IP Address:10.43.0.1, IP Address:127.0.0.1, IP Address:172.30.240.163, IP Address:172.30.240.25, IP Address:172.30.241.154, IP Address:172.30.241.96, IP Address:192.168.0.1, IP Address:44.236.77.201, IP Address:52.11.74.180, IP Address:52.24.177.17, IP Address:52.25.235.43, IP Address:52.41.143.127, IP Address:54.188.29.57, IP Address:0:0:0:0:0:0:0:1

[brandond]

To reset the SANs, you can use kubectl delete secret -n kube-system k3s-serving to delete the dynamiclistener secret, delete the cached copy at/var/lib/rancher/k3s/server/tls/dynamic-cert.json from all server nodes, and then restart k3s on the servers.

There is not currently any way to filter the CNs that are added to the cert; but there is a way to prevent additional certs from being added once you're happy with what it has on it - see the discussion at #7312 (comment)

[cheddarwhizzy]

Thank you! Unfortunately the control plane isn't accessible via kubectl due to the tls: handshake message length.. message.

kubectl get secret
Unable to connect to the server: tls: handshake message of length 70096 bytes exceeds maximum of 65536 bytes

You got any other tricks up your sleeve?

[cheddarwhizzy]

Tried deleting the secret from etcd as well - no dice

rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json

ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl  del -- "/registry/secrets/kube-system/k3s-serving"

systemctl restart k3s

I do have S3 etcd backups I could attempt to recreate the cluster and restore the snapshot, but I didn't have any luck last time attempted.

[cheddarwhizzy]

Scratch last post, the etcd stuff worked.. Guess I didn't need the --prefix flag for etcdctl

[brandond]

You can also connect directly to the apiserver, which has a different cert but only listens on localhost, and on a different port:
kubectl --server=https://localhost:6444 delete secret -n kube-system k3s-serving

[Chewie]

Facing this issue, I am unable to clean up the stuffed entries. Whenever I delete the k3s-serving secret, it gets recreated immediately, and also regenerates the dynamic-cert.json. I've tried deleting them in various orders, but the stuffed sans keep reappearing.

Is there a way to work around this?

[brandond]

Delete the file from all nodes, then delete the secret, then restart the nodes.

[Chewie]

When I tried that, deleting the secret would immediately recreate both the secret and the file.

However, I did end up with a ugly way to solve this issue: since it was a single node system with sqlite storage, I stopped k3s and deleted the entry directly in the sqlite db, along with the file as usual, and this time upon restarting the service a new cert was generated.

Cheers!