Resolved
After 3 days, 8 hours, and 36 seconds

We have confirmed that the service is fully restored. We will continue to closely monitor this situation for any re-occurrence. This incident is now closed.

Monitoring
After 3 days, 3 hours, and 57 minutes

Live data is caught up and has been processing since early this morning. Backlog and Hosted Reporting data continues to process and has approximately 1 hour remaining to complete.

We continue to closely monitor the reporting service and we will provide an update as soon as more information becomes available.

Identified
After 3 days and 57 minutes

The reporting service is processing live data and the Sync API is functional. We are still working through the backlog of historical data. Hosted reporting processing will begin once the backlog is cleared.

We will provide an update as soon as more information becomes available.

Identified
After 2 days, 10 hours, and 40 minutes

We are processing live data again and the Sync API is functional now. We still have the backlog of data from the past couple days to work through. Until this completes we won't have all the logs available and hosted reporting will not be functional. We are continuing to scale and monitor the system with the hope of being caught up and functional again by US morning.

We will provide an update as soon as more information becomes available.

Identified
After 1 day, 22 hours, and 2 minutes

A fix has been applied and we are now starting to work through the backlog of log data. Customers should begin to see data start trickling in, however hosted reporting data will not be processed until the backlog queue is cleared. We will continue to monitor and add resources as necessary to process the data as quickly as possible.

We will provide an update as soon as more information becomes available.

Identified
After 1 day, 17 hours, and 51 minutes

We continue to work on a fix. Customers may continue to experience a delay in accessing reporting and API events.

We will provide an update as soon as more information becomes available.

Identified

We are aware of customers in the US region experiencing a delay in accessing reporting and API events. Live data and reporting data will become available once the issue is resolved. In addition, customers using the Sync API will have been receiving empty zip files since 00:00 UTC on June 11, 2021. This will continue until the repair completes and log ingestion catches up. It is not a data loss event, however customers will need to manually re-request the data for the missing hours using a Sync API call with token=none and start date as 00:00 UTC June 11, 2021.

We have identified the cause of this issue and are working on a fix expected to complete in the next 24-36 hours. No data is lost, this is only impacting data reaching reports and API feeds. Once completed all data should be available with no action necessary. Please note that all other aspects of our Web Security Service remain functional.

We will provide an update as soon as more information becomes available.

Began at:

Affected components
  • Web Security Service
    • Portal & Reporting
      • Log Transfer Services
      • Reporting Services and Portal Dashboards
      • API Services