CARBON BLACK CLOUD - DATA FORWARDER - US

At approximately 9:04am EDT on Sep 12, 2023, a change to the CBC Data Forwarder codebase was deployed that broke data forwarder configurations for approximately 7% of Forwarder customers that had updated their Data Forwarder configurations within the last ~3 months.

This breakage did not trigger our usual automated alerting mechanisms and wasn’t detected until approximately 10:27am EDT on Sep 13, 2023. The engineering team was quickly engaged and deployed a fix at 13:51pm EDT.

For the affected customers, there could be up to 2 hours of permanent data loss for the data that flowed through Data Forwarder shortly after the change that broke customer configuration. The rest of the data that was affected during that time will be replayed into the affected customers s3 buckets. No further action by customers is required.

Our team is committed to ensuring that the set of changes that lead to this unfortunate outcome are thoroughly investigated and proper corrective and preventive actions are put into place to be more resilient in the future.

At approximately 9:04am EDT on Sep 12, 2023, a change to the CBC Data Forwarder codebase was deployed that broke data forwarder configurations for approximately 7% of Forwarder customers that had updated their Data Forwarder configurations within the last ~3 months.

This breakage did not trigger our usual automated alerting mechanisms and wasn’t detected until approximately 10:27am EDT on Sep 13, 2023. The engineering team was quickly engaged and deployed a fix at 13:51pm EDT.

For the affected customers, there could be up to 2 hours of permanent data loss for the data that flowed through Data Forwarder shortly after the change that broke customer configuration.

The rest of the data that was affected during that time will be replayed into the affected customers s3 buckets. No further action by customers is required.

Our team is committed to ensuring that the set of changes that lead to this unfortunate outcome are thoroughly investigated and proper corrective and preventive actions are put into place to be more resilient in the future.

A fix has been implemented and we are monitoring the results.

We have a fix ready. We will update when deployed.

We have found an erroneous code change from yesterday that the team believes to be the cause of the issue. We are working on a fix now.

Description: The CB team is investigating a potential issue with the CBC Data Forwarder. Our engineering teams are engaged, and we will provide a progress update within the next 60 minutes.

Customer Impact: Customers might experience delays with Data Forwarder sending events, alerts, or watchlist hits to customer configured S3 Buckets.

Security Impact: Security efficacy is not compromised due to this issue.

We are working diligently to restore the service back to normal performance. We thank you for your patience as we identify and resolve the issue.

VMware Carbon Black Team