Resolved
After 4 hours and 51 minutes

After closely monitoring both services, we are pleased to declare full service restoration.

We thank you for your patience while we addressed the issue and apologize for any inconvenience this may have caused. This will be the final update for this issue.

Monitoring
After 4 hours and 3 minutes

All email queues have been cleared and customers will no longer experience mail delays. Track and Trace results once again are available out of Virginia data-center. As previously communicated, Web service is also functioning as per normal.

Any customer still experiencing an issue is requested to contact our technical support team at your earliest convenience.

We are closely monitoring service and will provide final update after 21:00 UTC.

Monitoring
After 3 hours and 24 minutes

All email servers are back in service and processing mail without any issues. A small number of servers are still delivering previously queued email which we expect to get completed in the next hour.

We continue to closely monitor email and web traffic and will provide next update once we have updated information.

Monitoring
After 2 hours and 34 minutes

Web Security.cloud servers are back in service out of Virginia data-center and customers should no longer experience web latency.

Email Security.Cloud servers are being put back in service in a controlled fashion. Please note that while this exercise lasts, customers may experience some queue buildup. Once servers are loaded balanced, remaining email traffic will be delivered as soon as possible.

Next update will be posted at 20:00 UTC.

Identified
After 2 hours and 5 minutes

Our engineering team have found the cause of this incident and currently implementing a workaround to restore service as soon as possible.

Next update will be posted after 30 minutes.

Investigating
After 1 hour and 36 minutes

Email.Cloud customers are experiencing delays sending and receiving emails. Web.Cloud customers experience web browsing latency.

Our engineers are currently rolling out some changes to attempt to reduce the impact.

Next update will be posted at 18:40 UTC.

Investigating
After 31 minutes

Our engineers continue investigation. Any new email is getting delivered without any issue however emails in retry will be delayed until we bring Virginia Data-center back in service. Web Security.Cloud customers may experience some latency as they use service out of secondary data-centers.

We are treating this as a high priority incident and will continue to regularly provide updates as they become available.

Investigating

As a result of some network issues out of Virginia data-center, our engineers have taken both email and web servers out of service and failed over services to alternate data-centers. Email and Web traffic is working as expected.

Clientnet TNT searches for emails passed via Virginia data-center will be unavailable at this time. Our engineering team is aware of the problem and are currently investigating.

We will make a further posting at 17:30 UTC or as soon as more information becomes available.

Began at:

Affected components