VMware SASE - Investigating VCO124 & VCO47 Orchestrator management services / UI may be inaccessible

The VMware SASE team has not encountered additional issues in monitoring and has implemented a fix to resolve this incident. Accordingly, this is the final notification for this incident which will now be closed.

Start Time: April 12, 2023 06:06 UTC End Time: April 15, 2023 14:11 UTC

Current Impact: None (issue resolved)

The VMware SASE team performed an emergency fix April 15th and VCO124 statistics are now available, VMware SASE team will monitor this during the next 4 hours prior closure.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 15, 2023 14:11 UTC

Current Impact: No current impact, statistics are available now.

Update - The VMware SASE team isolated the cause of the file backlog to client device migration, which caused a higher amount of files to be processed. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease at a lower pace in the final stages for VCO124. The VMware SASE team will perform an emergency fix April 15th and completed successfully.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~97,9%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 15:30 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on April 15th 15:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics on VCO124 for environment devices.

Update - The VMware SASE team isolated the cause of the file backlog to client device migration, which caused a higher amount of files to be processed. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease at a lower pace in the final stages for VCO124. The VMware SASE team will perform an emergency fix April 15th at 05:00 UTC to increase recovery speed, work started and is currently in progress.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~97,5%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 07:30 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on April 15th 08:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics on VCO124 for environment devices.

Update - The VMware SASE team isolated the cause of the file backlog to client device migration, which caused a higher amount of files to be processed. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease at a lower pace in the final stages for VCO124. The VMware SASE team will perform an emergency fix April 15th at 05:00 UTC to increase recovery speed.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~97,5%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 06:00 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on April 15th 06:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics on VCO124 for environment devices.

Update - The VMware SASE team isolated the cause of the file backlog to client device migration, which caused a higher amount of files to be processed. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease at a lower pace in the final stages for VCO124. The VMware SASE team will perform an emergency fix April 14th at 03:00 UTC to increase recovery speed.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~97%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 04:00 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 15th April 05:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics on VCO124 for environment devices.

Update - The VMware SASE team isolated the cause of the file backlog to client device migration, which caused a higher amount of files to be processed. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease at a lower pace in the final stages for VCO124. The VMware SASE team is reviewing this and will perform an emergency fix to increase recovery speed.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~97%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 14, 2023 24:00 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 14th April 24:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics on VCO124 for environment devices.

Update - The VMware SASE team isolated cause to client device migration that caused a higher amount of files producing file backlog. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease at a lower pace in the final stages for VCO124, VMware SASE team is reviewing the behavior.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~95%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 14, 2023 20:26 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 14th April 22:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

Update - The VMware SASE team isolated cause to client device migration that caused a higher amount of files producing file backlog. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease as expected.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~92%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 14, 2023 16:53 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 14th April 18:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

The VMware SASE team isolated cause to client device migration that caused a higher amount of files producing file backlog. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease as expected.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~65%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 12:00 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 14th April 12:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

The VMware SASE team isolated cause to client device migration that caused a higher amount of files producing file backlog. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease as expected.

VCO47 backlog processing completed.

Start Time: April 12, 2023 06:06 UTC End Time: April 14, 2023 04:26 UTC

VCO124 Progress of backlog completion : ~50%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 12:00 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 14th April 08:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

The VMware SASE team isolated cause to client device migration that caused a higher amount of files producing file backlog. As the data points are processed, statistics will become available for our partners and customers. The backlog count continues to decrease as expected.

VCO47 Progress of backlog completion : ~90%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 14, 2023 12:00 UTC

VCO124 Progress of backlog completion : ~42%

Start Time: April 12, 2023 06:06 UTC End Time ETA: April 15, 2023 12:00 UTC

The VMware SASE team continues to actively work on this incident, next progress update will be provided on 14th April 04:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

Update - The VMware SASE team is still working towards mitigation of impact. Engineers have implemented a strategy to decrease the backlog of transactions that are possibly impacting the ability to view statistics in the SD-WAN environments on the affected VCO's. The backlog is currently estimated to be completed before 1200 UTC on 14 April. As the data points are processed, they will become available for our partners and customers.

Start Time: April 12, 2023 06:06 UTC End Time: To be determined

Next progress update will be provided on 13th April 22:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

The VMware SASE team is still working towards mitigation of impact. Engineers have implemented a strategy to decrease the backlog of transactions that are possibly impacting the ability to view statistics in the SD-WAN environments on the affected VCO's. The backlog is currently estimated to be completed before 1200 UTC on 14 April. As the data points are processed, they will become available for our partners and customers.

Start Time: April 12, 2023 06:06 UTC End Time: To be determined

Next progress update will be provided on 13th April 14:00 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

The VMware SASE team is still working towards mitigation of impact. Engineers have implemented a strategy to decrease the backlog of transactions that are possibly impacting the ability to view statistics in the SD-WAN environments on the affected VCO's. The backlog is currently estimated to be completed before 1200 UTC on 14 April. As the data points are processed, they will become available for our partners and customers.

Start Time: April 12, 2023 06:06 UTC End Time: To be determined

Next progress update will be provided on 13th April 0400 UTC or sooner if more information is available.

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices.

The VMware SASE team is actively working towards mitigating current impact, further updates will be provided shortly.

Start Time: April 12, 2023 06:06 UTC End Time: To be determined

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices. (No traffic impact)

Service impact has been identified. The VMware SASE team is still investigating and working to mitigate impact. Further updates will be provided shortly.

Start Time: April 12, 2023 06:06 UTC End Time: To be determined

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices

The VMware SASE team has become aware of a potential service event and is gathering more information. Engineering resources have been engaged and an update will be provided shortly.

Start Time: April 12, 2023 06:06 UTC End Time: To be determined

Current Impact: Customer may not be able to access Orchestrator statistics for environment devices