Resolved
Resolved

We have monitored the service for stability and this incident shall now be closed.

Monitoring

The backlog is fully cleared and all the new alerts are processing without any delays. We will now monitor the service for stability and provide updates as soon as new information becomes available.

Update

We have restarted the impacted services and the processing of the alert backlog is now underway. Once the backlog is fully cleared, the queue of new alerts will begin processing. We are closely monitoring the system to ensure a smooth transition back to real-time delivery.

Update

We are seeing significant improvement as new alerts are now processing without delay. Our systems are successfully delivering current queue alerts, while the backlog from the incident period is currently being re-processed. Our team continues to finalize the fix and monitor the recovery to ensure all alerts are successfully delivered.

Identified

We have identified the cause of the issue and have implemented a fix. The alerts backlog is currently processing. We will provide updates as soon as new information becomes available.

Investigating

We are aware of an issue in our Alerts Pipeline which is currently affecting all customers on the US infrastructure. Processing of the new alerts is delayed. The new alerts will not be available on the Console. However customers will be able to search/manage existing alerts. Sensors will continue to monitor and prevent threats in your environment.

Our engineering teams are engaged and actively investigating the issue. We will continue to provide updates as more information becomes available.

Began at:

Affected components
  • Carbon Black
    • ENDPOINT STANDARD / PREVENTION
      • US 1
        • Event Processing
      • US 2
        • Event Processing
      • US 5
        • Event Processing