Email Security.cloud Track & Trace slowness and incomplete results

Resolved
Resolved

We have monitored the service for stability and this incident shall now be closed.

Monitoring

We have implemented a fix on the remaining TNT infrastructure. We will now monitor the service for stability and provide updates as soon as new information becomes available.

Update

We have deployed the fix to production TNT server and we are already seeing improved response times and search results. The fix is applied to one of the two production servers as we continue to monitor performance. We will then apply the fix to the remaining TNT infrastructure hopefully later this week. Customers should no longer experience performance issues.

We will provide an update as soon as more information becomes available.

Update

We are going to start releasing the fix on TNT servers April 7 onwards with a completion date of April 17, 2023. Our testing shows significant improvement in index build times and search query results. We are expecting customers to start seeing improvements the week of April 10, 2023 as we continue the deployment in a staggered manner.

We will provide an update as soon as more information becomes available.

Update

We have not seen performance improvements after increasing server memory (temporary fix). We continue to perform testing for the permanent fix and can confirm that we are seeing positive results so far. The deployment cycle for this fix is expected to take 2 weeks for completion. We will provide more updates once the testing is complete.

Update

A temporary fix has been deployed and we are currently monitoring the results and performance. In addition, we continue to work on a permanent fix.

We will provide an update as soon as more information becomes available.

Update

We are aiming to rollout the temporary fix by the end of NAM day tomorrow March 31, 2023. The work on the permanent fix is in progress and we will provide additional updates once the testing is completed.

Update

We continue to work on both the temporary and the permanent fix. Some customers may continue to experience issues with Clientnet Track and Trace searches for email, resulting in slowness or partial results.

We will provide an update as soon as more information becomes available.

Identified

We have identified the cause of the issue and are working on a permanent fix. The fix is currently being tested and we will provide deployment dates/status once the testing is completed. We are also working on a temporary solution but it is unclear as to how much that will improve the overall T&T performance.

We will provide updates as new information becomes available.

Update

Our engineers continue to investigate this issue. We have some leads but the root cause is yet to be identified. We will provide an update as soon as more information becomes available.

Update

We continue to investigate this issue. We will provide an update as soon as more information becomes available.

Update

We continue to investigate this issue. We will provide an update as soon as more information becomes available.

Investigating

We are aware of a subset of customers currently experiencing issues with Clientnet Track and Trace searches for email, resulting in slowness or partial results. Our engineering team is currently investigating the issue. Please note that there is no impact to delivery and security scanning of your email traffic, and all other aspects of our services remain functional. We will provide an update as soon as more information becomes available.

Began at:

Affected components