test: Shutdown Kafka Brokers before ZooKeeper/KRaft Controllers

[NickLarsenNZ]

I noticed tests would sometimes fail due to timeouts deleting the namespace. It turned out ZooKeeper pods were shut down before Kafka Brokers, and the brokers were trying to communicate with ZooKeeper (or the KRaft controller).

Caution

This currently fails for zookeeperless clusters. After brokers have been scaled down and we attempt to scale down controllers, we see this error:

Warning  BuildConfigMap  2s (x22 over 107s)  kafkacluster.kafka.stackable.tech  failed to build configmap: no Kraft controllers found to build

This might depend on #955

Last test results

--- FAIL: kuttl/harness/configuration_kafka-latest-3.9.1_openshift-false (312.10s)
--- FAIL: kuttl/harness/logging_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false (511.70s)
--- FAIL: kuttl/harness/operations-kraft_kafka-kraft-3.9.1_openshift-false (587.05s)
--- FAIL: kuttl/harness/operations-kraft_kafka-kraft-4.1.1_openshift-false (649.95s)
--- FAIL: kuttl/harness/smoke-kraft_kafka-kraft-3.9.1_openshift-false (553.01s)
--- FAIL: kuttl/harness/smoke-kraft_kafka-kraft-4.1.1_openshift-false (759.79s)
--- FAIL: kuttl/harness/upgrade_upgrade_old-3.9.1_upgrade_new-4.1.1_use-client-tls-false_use-client-auth-tls-false_openshift-false (396.50s)
--- FAIL: kuttl/harness/upgrade_upgrade_old-3.9.1_upgrade_new-4.1.1_use-client-tls-false_use-client-auth-tls-true_openshift-false (398.25s)
--- FAIL: kuttl/harness/upgrade_upgrade_old-3.9.1_upgrade_new-4.1.1_use-client-tls-true_use-client-auth-tls-false_openshift-false (395.12s)
--- FAIL: kuttl/harness/upgrade_upgrade_old-3.9.1_upgrade_new-4.1.1_use-client-tls-true_use-client-auth-tls-true_openshift-false (413.45s)
--- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.1_zookeeper-latest-3.9.4_openshift-false (55.44s)
--- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false (38.70s)
--- PASS: kuttl/harness/kerberos_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-cluster-internal (127.73s)
--- PASS: kuttl/harness/kerberos_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-stable (135.15s)
--- PASS: kuttl/harness/kerberos_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (149.73s)
--- PASS: kuttl/harness/kerberos_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-cluster-internal (140.33s)
--- PASS: kuttl/harness/kerberos_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-stable (174.00s)
--- PASS: kuttl/harness/kerberos_kafka-3.9.1_zookeeper-latest-3.9.4_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (144.87s)
--- PASS: kuttl/harness/opa_kafka-latest-3.9.1_zookeeper-latest-3.9.4_opa-latest-1.12.3_use-opa-tls-false_openshift-false_krb5-1.21.1 (124.94s)
--- PASS: kuttl/harness/opa_kafka-latest-3.9.1_zookeeper-latest-3.9.4_opa-latest-1.12.3_use-opa-tls-true_openshift-false_krb5-1.21.1 (146.65s)
--- PASS: kuttl/harness/smoke_kafka-3.9.1_zookeeper-3.9.4_use-client-tls-false_openshift-false (79.51s)
--- PASS: kuttl/harness/smoke_kafka-3.9.1_zookeeper-3.9.4_use-client-tls-true_openshift-false (89.54s)
--- PASS: kuttl/harness/tls_kafka-3.9.1_zookeeper-latest-3.9.4_use-client-tls-false_use-client-auth-tls-false_openshift-false (46.43s)
--- PASS: kuttl/harness/tls_kafka-3.9.1_zookeeper-latest-3.9.4_use-client-tls-false_use-client-auth-tls-true_openshift-false (196.36s)
--- PASS: kuttl/harness/tls_kafka-3.9.1_zookeeper-latest-3.9.4_use-client-tls-true_use-client-auth-tls-false_openshift-false (228.95s)
--- PASS: kuttl/harness/tls_kafka-3.9.1_zookeeper-latest-3.9.4_use-client-tls-true_use-client-auth-tls-true_openshift-false (192.11s)