VMware SASE - Identified: Multiple regions CWS Impacted

The VMware SASE team has not encountered additional issues in monitoring and has implemented a fix to resolve this incident. Accordingly, this is the final notification for this incident which will now be closed.

Start Time: May 24, 2023 21:26 UTC End Time: May 24, 2023 23:10 UTC

Current Impact: None (issue resolved)

The VMware SASE team has rolled back the logging change and issue is no longer observed.

VMware team will monitor for an hour and proceed with marking this as resolved.

Start Time: May 24, 2023 21:26 UTC End Time: May 24, 2023 23:10 UTC

Current Impact: None (issue mitigated and being monitored)

The VMware SASE team identified that a recently implemented change in logging is causing impact on the Ingress traffic for VNI configuration. VMware is currently evaluating temporarily disabling the log enhancer feature as a temporary measure meanwhile the investigation continues.

VMware teams are actively engaged on recovery activities and next update will be provided in the next hour or earlier if available.

List of identified components affected but not limited to: vco135-usvi1 vco160-usca1 vco111-usca1 vco206-fra1 vco309-syd1 vco11-usvi1 vco84-usvi1 vco61-usvi1 vco115-usca1 vco212-fra1 vco33-usvi1 vco21-usvi1 vco214-fra1 vco8-fra1 vco75-usvi1 vco301-syd1 vco128-usvi1 vco124-usca1 vco58-usvi1 vco109-usca1 vco35-usvi1 vco161-usca1 vco211-fra1 vco125-usca1 vco150-usca1 vco134-usvi1 vco207-fra1 vco129-usvi1 vco7-sin1 vco87-usvi1 vco57-usvi1 vco31-usvi1 vco6-jpn1 vco133-usvi1 vco203-fra1 vco351-bom1 vco402-yul1 vco97-usvi1 vco18-usvi1 vco47-usvi1 vco401-yul1 vco20-usca1 vco12-usvi1 vco252-lhr3 vco22-fra1 vco178-usca1

Start Time: May 24, 2023 21:26 UTC End Time: To be determined

Current Impact: Single sign on configuration may not be changed and new customer onboarding may not be functional. Other management plane services may experience intermittent issues.

The VMware SASE team has identified that there are intermittent failures in the health checks of VCO. These failures are occurring during parallel checks of VNI services, resulting in too many API requests and error messages. The behavior is inconsistent, and the service-level CWS connections are connecting to a centralized service.

VMware teams are actively engaged on recovery activities and next update will be provided in the next hour or earlier if available.

List of identified components affected but not limited to: vco135-usvi1 vco160-usca1 vco111-usca1 vco206-fra1 vco309-syd1 vco11-usvi1 vco84-usvi1 vco61-usvi1 vco115-usca1 vco212-fra1 vco33-usvi1 vco21-usvi1 vco214-fra1 vco8-fra1 vco75-usvi1 vco301-syd1 vco128-usvi1 vco124-usca1 vco58-usvi1 vco109-usca1 vco35-usvi1 vco161-usca1 vco211-fra1 vco125-usca1 vco150-usca1 vco134-usvi1 vco207-fra1 vco129-usvi1 vco7-sin1 vco87-usvi1 vco57-usvi1 vco31-usvi1 vco6-jpn1 vco133-usvi1 vco203-fra1 vco351-bom1 vco402-yul1 vco97-usvi1 vco18-usvi1 vco47-usvi1 vco401-yul1 vco20-usca1 vco12-usvi1 vco252-lhr3 vco22-fra1 vco178-usca1

Start Time: May 24, 2023 21:26 UTC End Time: To be determined

Current Impact: Single sign on configuration may not be changed and new customer onboarding may not be functional. Other management plane services may experience intermittent issues.