Securlets Service Disruption

We have confirmed the service is fully restored. This incident is now resolved.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, finding & remediating of exposures, and logs exports with SIEM agents. We are aware of delays in content re-scan tasks and Detect incidents on some tenants and expect them to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, finding & remediating of exposures, and logs exports with SIEM agents. We are aware of delays in content re-scan tasks and Detect incidents on some tenants and expect them to clear soon.

We updated the scheduling of infrastructure tasks to better control the concurrency and contention between them. We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, finding & remediating of exposures, and logs exports with SIEM agents. We are aware of delays in content re-scan tasks and Detect incidents and these are expected to clear soon.

We updated the scheduling of infrastructure tasks to better control the concurrency and contention between them. We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, finding & remediating of exposures, and logs exports with SIEM agents. We are aware of delays in content re-scan tasks and Detect incidents and these are expected to clear soon.

The next update will be posted on September 9, 2018 at 01:00 UTC. Moving forward we will provide updates every 24 hours or as soon as new information becomes available.

We continue to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Backlog of log exports from SIEM agents should be cleared. We are aware of delays in content re-scan tasks and Detect incidents and these are expected to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Backlog of log exports from SIEM agents should be cleared. We are aware of delays in content re-scan tasks and Detect incidents and these are expected to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Backlog of log exports from SIEM agents should be cleared. We are aware of delays in content re-scan tasks and Detect incidents and these are expected to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. We are aware of delays in log exports using SIEM agents and expect it to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. We are aware of delays in log exports using SIEM agents and expect it to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. We are aware of delays in log exports using SIEM agents and expect it to clear soon.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. For most of the customers, backlog of events should have cleared by now and for others it is expected to clear soon.

The next update will be posted at 23:00 UTC. Moving forward we will provide updates every 8 hours or as soon as new information becomes available.

We are continuing to monitor the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Some customers may see some backlog of events that are still pending, but are expected to clear soon.

We are monitoring the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Some customers may see some backlog of events that are still pending, but are expected to clear soon.

We are monitoring the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Some customers may see some backlog of events that are still pending, but are expected to clear soon.

A generalized fix to prevent creation of unnecessary data indices by the application layer has been applied as a patch. We are monitoring the database cluster health and no issues have been seen so far.

Securlet functionality is operational at this time including reporting of user activity events, content scanning, and finding & remediating of exposures. Some customers may see some backlog of events that are still pending, but are expected to clear soon.

We have applied the generalized fix for the application level cause, specifically to prevent creation of unnecessary data indices. We are monitoring the database cluster health and no issues have been seen so far. We will continue to monitor the system to ensure system has fully recovered.

At this time, Securlet customers can continue to access historical data. While new data, exposures, and remediations will continue to see some delays, the backlog is expected to be smaller. We will provide further updates as new information is discovered.

We will be soon applying the generalized fix for the application level cause, specifically to prevent creation of unnecessary data indices.

At this time, Securlet customers can continue to access historical data. While new data, exposures, and remediations will continue to see some delays, the backlog is expected to be smaller. We will provide further updates as new information is discovered.

The Securlet service is operational at the same capacity as the last update. We are monitoring the database cluster health and no issues have been seen so far.

We are testing the generalized fix for the application level cause, specifically to prevent creation of unnecessary data indices. Investigation is still on for any other potential application level causes.

At this time, Securlet customers can continue to access historical data. While new data, exposures, and remediations will continue to see some delays, the backlog is expected to be smaller. We will provide further updates as new information is discovered.

We have further increased the capacity of the Securlet resources since the last progress update. We are monitoring the database cluster health at this higher Securlet capacity and no issue has been seen so far.

We are testing the generalized fix for the application level cause, specifically to prevent creation of unnecessary data indices. Investigation is still on for any other potential application level causes.

At this time, Securlet customers can continue to access historical data. While new data, exposures, and remediations will continue to see some delays, the backlog is expected to be smaller. We will provide further updates as new information is discovered.

We have increased the capacity of the Securlet resources since the last progress update, and no significant bursts have been observed on database load. We are continuing to monitor the database cluster health at this higher Securlet capacity.

We are testing the generalized fix for the application level cause, specifically to prevent creation of unnecessary data indices. Investigation is still on for any other potential application level causes.

At this time, Securlet customers can continue to access historical data. While new data, exposures, and remediations will continue to see delays, the backlog is expected to be smaller. We will provide further updates as new information is discovered.

Database rebalancing, and archiving & purging of aged data has been ongoing, and has so far significantly increased available resources, and will allow the database to better handle future bursty load patterns. This effort is ongoing and will continue through the restore process.

We are working on a more generalized fix for the application level causes, specifically to prevent creation of unnecessary data indices. Investigation is still on for any other potential application level causes.

The plan is to ramp up the Securlet services in a controlled fashion, while observing the database health.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We are continuing to operate the Securlet service at a significantly reduced capacity to allow for the database re-balancing to progress. We are also archiving and purging aged data in parallel to increase available database resources.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We are continuing to operate the Securlet service at a significantly reduced capacity to allow for the database re-balancing to progress. We are also archiving and purging aged data in parallel to increase available database resources.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We are continuing to operate the Securlet service at a significantly reduced capacity to allow for the database rebalancing to progress. We are also archiving and purging aged data in parallel to increase available database resources.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We are continuing to rebalance across all the nodes in the database cluster, including the new ones added recently. In order to allow this to be done more aggressively, we have now reduced the Securlet capacity.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We are continuing to rebalance across all the nodes in the database cluster, including the new ones added recently. In order to allow this to be done more aggressively, we have now reduced the Securlet capacity.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We continue to investigate the database tasks contributing to Securlet service degradation. The database cluster is being rebalanced to more evenly distribute the bursty load across the cluster. We are keeping the system at reduced capacity to do more aggressive database rebalancing.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We continue to investigate the disruptions affecting the Securlet service. We are keeping the Securlet system at reduced capacity to allow maintenance to progress.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays. We will provide further updates as new information is discovered.

We are continuing to monitor the recovery of Securlet service and work on database maintenance activities. Service has been stable and rebalancing of the database cluster is continuing as planned.

At this time, some Securlet customers will continue to see delays for, exposures, remediations and on-demand scans

Securlet resources have been increased to 50% of pre-incident capacity level, and no significant bursts have been observed on database load. No new application level causes for the database load bursts have been found, and we continue to monitor to find any potential ones.

In parallel, rebalancing of the database cluster is continuing as an ongoing activity.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays for some.

We will provide further updates as new information is discovered.

The fixes that were applied to the initially discovered application level issues causing the database load bursts were found effective and have now been applied more broadly across all instances.

Additional capacity has been added to the Securlet resources since the last progress update. At this increased capacity, database load is being monitored for any other potential application level causes. In parallel, rebalancing of the database cluster is continuing as an ongoing activity. We will provide further updates as new information is discovered.

At this time, Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays.

Our investigation into the disruption to the Securlet service continues. We are continuously evaluating and addressing the bursty load seen in the database cluster in an effort to restore the service.

We have increased the available resources for the Securlet service and are evaluating the changes.

We will provide further updates as new information is discovered.

During this incident Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays.

Our investigation into the issues disrupting Securlet service continues. In an effort to remediate this disruption, the database cluster is being rebalanced to more evenly distribute the bursty load across the cluster. This effort is ongoing and will continue through the restoration process.

Additionally, a few application level issues that are contributing to the database load bursts were identified and patches were applied to them. We continue our investigation for all other potential application level causes.

Our plan is to increase available resources for the Securlet service in a controlled fashion, while observing the database health as we continue our investigation.

During this incident Securlet customers can continue to access historical data while new data, exposures, and remediations will continue to see delays.

We continuing to investigate the cause behind database performance issues and work towards a resolution.

We will provide further updates as new information is discovered.

We are continuing to investigate the cause behind database performance issues.

We will provide further updates as new information is discovered.

We are continuing to investigate the cause behind database performance issues.

We will provide further updates as new information is discovered.

We are continuing to investigate the cause behind database performance issues.

We will provide further updates as new information is discovered.

We are continuing to experience database performance issues. Securlet functionality has been temporarily paused. Access to logs and reporting of new events, exposures, and remediations will be delayed.

We are continuing to investigate this issue. We will provide further updates as new information is discovered.

A solution has been implemented and we are monitoring the system closely. Securlets are now fully functional. Some customers may notice delays with reporting of new events, exposures, and remediations. However we expect this to clear in the next few hours.

We continue to work on a fix to resolve the previously reported concerns. We will provide further updates as they become available.

Securlet functionality has been temporarily paused to alleviate a database issue. Customers can continue to access old events but reporting of new events, exposures, and remediations will be delayed. We are continuing to work on a fix to resolve this issue. We will provide further updates as new information is discovered.

We are facing database outage at the moment and the Securlet service is down. We are continuing to work on a fix to resolve this issue. We will provide further updates as new information is discovered.

We are continuing to work on a fix for the Securlets issue. We will provide further updates as new information is discovered.

We are continuing to remediate the previously reported issue. We will provide further updates as new information is discovered.

We are aware customers may be experiencing issues with certain functions of Securlets. We have identified a potential cause for the issue and are in process of deploying a solution. We will provide further updates as new information is discovered.