VMware Tanzu Mission Control Disconnected Clusters Issue

We have monitored the service for stability and this incident shall now be closed.

We have implemented a fix. We will now monitor the service for stability and provide updates as soon as new information becomes available.

We continue to work on a fix. We will provide updates as new information becomes available.

We continue to work on a permanent fix. As a workaround, If you see a lot of orphaned cluster-auth-pinniped-impersonation-proxy-token-xxxxx secrets in the vmware-system-tmc namespace, please run the following command to clean them. We are working on a fix to clean up the secrets without manual intervention. We will provide further updates as more information becomes available.

kubectl get secrets -o=jsonpath='{.items[?(@.metadata.annotations.kubernetes.io/service-account.name=="cluster-auth-pinniped-impersonation-proxy")].metadata.name}' | sed 's/ /\n/g' | awk '/.*/{print "kubectl delete secret "$0;next}1' | bash

We have identified the cause of the issue and working on a fix. We will provide updates as new information becomes available.

VMware Tanzu Mission Control customers on Kubernetes cluster versions 1.23 or lower might be experiencing some unexpected behavior on their clusters. Some of these clusters could be disconnected. Our engineering teams are engaged and actively working to mitigate the issue. All other aspects of the service remain operational.