Resolved

The previous reported concern with email delays has been resolved. The service is fully restored and there is no longer an impact to customers. This incident is resolved.

Update

The service remains fully operational and customers should continue to see mail flowing normally. We continue to closely monitor the email traffic to ensure the service is fully restored.

Monitoring

As of June 11, 2019 at 03:34 UTC, existing email queues have cleared for all regions and the service is fully operational. Customers should see mail flowing normally. We will continue to closely monitor the email traffic to ensure smooth service operation.

Update

We are continuing to work on this issue with the highest priority. Here is the latest update:

EMEA Infrastructure: As of 22:35 UTC, all existing email queues have cleared, and the service is back to operational. Customers should now see mail flowing normally. We are continuing to monitor the email traffic and will provide further updates as new information is discovered.

NAM Infrastructure: The majority of new emails are being delivered without any delay. Existing queued emails continue to be delivered at a reduced rate. A small subset of customers may continue to experience delays sending or receiving new and existing emails.

We will provide further updates as more information becomes available.

Update

We are continuing to work on this issue with the highest priority. Here is the latest update:

The majority of new emails are being delivered without any delay. Existing queued emails continue to be delivered at a reduced rate. A subset of customers may continue to experience delays sending or receiving new and existing emails.

We will provide further updates as more information becomes available.

Update

Post introduction of additional processing capacity, we are starting to see improvements with new emails processing. Existing queued emails continue to be delivered but at a reduced rate. In the meantime, our engineers continue to manually load balance email traffic and have taken impacted infrastructure out of service where possible. Some customers may continue to experience delays sending or receiving emails.

We are continuing to work on this issue with the highest priority and will provide further updates as more information becomes available.

Identified

We are making positive progress to fix the issue. Previously impacted mail servers are now processing queued emails. In parallel, our engineers are also working on bringing additional servers online to handle the traffic.

Some customers will already be seeing large amounts of the backlogged mail being delivered. Please note that there is no impact to protection services, and all other aspects of our services remain functional.

We will provide further updates as soon as new information becomes available.

Monitoring

We are progressing in our efforts to resolve the reduced delivery rate for queued emails. Customers may continue to experience delays sending or receiving emails.

We will provide further updates as more information becomes available.

Update

Existing queued emails are being delivered but at a reduced rate. Our engineers are looking in to the cause behind this and are actively engaged as we work towards full-service restoration. Customers may continue to experience delays sending or receiving emails.

We will provide further updates as more information becomes available.

Update

As of approximately 14:10 UTC, we have applied mitigation to restore the service. Queued emails have started getting delivered.

We will provide further updates as more information becomes available.

Identified

Our engineers have identified a possible cause of the issue and we are now focused upon resolution efforts. Queued emails will be processed successfully once service is restored.

Our engineers continue to be actively engaged as we work towards service restoration. We will provide further updates as more information becomes available.

Investigating

We are aware of customers provisioned on NAM and EMEA based infrastructure experiencing delays sending or receiving emails since approximately 12:15 UTC. We will deliver queued email following our retry schedule once service is restored.

We are handling this issue as a high priority and have engaged all appropriate teams at Symantec. Please note that there is no impact to protection services, and all other aspects of our services remain functional.

We will provide further updates as soon as more information becomes available.

Began at:

Affected components