Resolved
Resolved
After 4 days, 19 hours, and 18 minutes

We have monitored the service for stability and this incident shall now be closed.

Monitoring
After 2 days, 3 hours, and 57 minutes

We can confirm that data ingestion is complete and the service is fully restored. During the incident timeline, some log data was not provided over the API. No log data was lost, but customers that wish to retrieve it must retrieve the missing data manually. Note that there may be a small amount of log duplication. Please see the following link containing steps to retrieve data:

https://knowledge.broadcom.com/external/article?articleId=244679

We will now monitor the service for stability and provide updates as soon as new information becomes available.

Identified
After 1 day, 4 hours, and 25 minutes

We continue to work on completing the ingestion of the delayed/missing data. We will provides updates as new information becomes available.

Identified
After 3 hours and 47 minutes

We have deployed a hotfix to all regions and we now appear to be successfully ingesting all data. However, we still have data that has not been ingested between the yesterday’s release and the fix just deployed. We are working on this remaining dataset and will provide our next update tomorrow June 23, 2022 at 20:00 UTC.

Identified
After 2 hours and 21 minutes

We are in the process of deploying a fix. We will provide an update as soon as more information becomes available.

Identified

We are aware of customers experiencing logs and reporting issues through Sync API, portal reporting and dashboards. Impacted customers may experience missing data due to ingestion delays. We have identified the cause of this issue and working on a fix.

Please note that all other aspects of our Web Security Service remain functional. We will provide an update as soon as more information becomes available.

Began at:

Affected components
  • Web Security Service
    • Portal & Reporting
      • Log Transfer Services
      • Reporting Services and Portal Dashboards
      • API Services