Email Delays in NAM Region

The previous reported concern with email delays has been resolved. The service is fully restored and there is no longer an impact to customers. This incident is resolved.

As of 14:45 UTC, we have rectified the underlying problem causing email delivery issue. Email queues are back to pre-incident levels and all new email is being processed and delivered without any delay. Customers should see mail flowing normally.

We are continuing to closely monitor the email traffic and will provide further updates as new information is discovered.

US-East Infrastructure: All new emails continue to deliver without any delay.

US-West infrastructure: A vast majority of queued emails have now been delivered. The remaining queued email will be delivered following our retry schedule.

The impacted US-West infrastructure remains out of service. We are closely working with our vendor to address the underlying cause. We will provide further updates as new information is discovered.

US-East Infrastructure: As of 05:45 UTC, all new email is being delivered without any delay.

US-West infrastructure: Queued email will be delivered following our retry schedule.

We are continuing to work on this issue as a highest priority and will provide further updates as new information is discovered.

US-East Infrastructure: As of 05:45 UTC, all new email is being delivered without any delay.

US-West infrastructure: Queued email will be delivered following our retry schedule.

We are continuing to work on this issue as a highest priority and will provide further update at 15:00 UTC, or as new information is discovered.

US-East Infrastructure: As of 05:45 UTC, all new email is being delivered without any delay.

US-West infrastructure: Queued email will be delivered following our retry schedule. We are continuing to work on this issue as a highest priority and will provide further update at 09:00 UTC, or as new information is discovered.

As of 4:57 UTC, the US-East Infrastructure has been brought back into service and is processing email without delay.

US-West Infrastructure: Some customers may continue to see delays sending and receiving email.

We are continuing to work on this issue as a highest priority and will provide further updates as more information becomes available.

We will provide our next update at 09:00 UTC, or as new information is discovered.

US-West infrastructure: Customers may continue to experience issues sending and receiving emails. Queued email will be delivered following our retry schedule, once service is restored.

US-East infrastructure: All queued emails have been delivered and US-East infrastructure remains out of service.

We are handling this issue as a high priority and have engaged all appropriate teams at Broadcom.

We will provide our next update at 05:00 UTC, or as new information is discovered.

We are aware of customers experiencing delays sending or receiving emails processed by US-West infrastructure. This is impacting both new and queued email.

The US-East infrastructure remains out of service and continues to process previously queued email.

We will provide our next update at 03:00 UTC, or as new information is discovered.

As of 23:30 UTC, the mail queues on US-East infrastructure continue to decrease and it is expected to clear in an hour. All new emails continue being processed by US-West infrastructure without any delays. We continue to closely monitor the remaining queued emails.

We will provide an update on April 17, 2020 at 02:00 UTC or as soon as more information becomes available.

Email Delay Issue: As of 22:30 UTC, the mail queues on US-East infrastructure continue to decrease. All new emails continue being processed by US-West infrastructure without any delays. We continue to closely monitor the remaining queued emails.

Email Delivery Issue: Between approximately 21:00 and 21:35 UTC, a very small subset of customer emails processed by mail tower T345 experienced a mail delivery issue. Email connections to this tower were presented with delivery errors. To remediate this issue, the impacted infrastructure has been taken out of service and there is no further customer impact. This issue is believed to be unrelated to the current mail delay issue.

We will provide an update at 23:45 UTC or as soon as more information becomes available.

As of 21:20 UTC, the mail queues on US-East infrastructure continue to decrease. All new emails continue being processed by US-West infrastructure without any delays.

We continue to closely monitor the remaining queued emails.

We will provide an update at 22:30 UTC or as soon as more information becomes available.

As of 19:53 UTC, the remaining impacted US-East infrastructure has been taken out of service. Queued emails are now being delivered at a much faster rate. All new emails are now being processed by US-West infrastructure without any delays.

We continue to closely monitor the remaining queued emails.

We will provide an update at 21:30 UTC or as soon as more information becomes available.

As of 19:15 UTC, we have taken half of the impacted US-East infrastructure out of service. This is to stop accepting any new emails and allow queued emails to deliver from US-East infrastructure. We are monitoring the queue closely to validate if this action helps with reducing mail queues. Note that US-West infrastructure continues to process emails without any issue.

Customers may continue to experience delays sending or receiving email. We will deliver queued email following our retry schedule once service is restored.

We will provide an update at 21:00 UTC or as soon as more information becomes available.

Our engineering team is actively investigating the cause behind the slow rate of mail delivery. Customers may continue to experience delays sending or receiving email. This issue is only impacting emails processed by US-East infrastructure. Any emails passing US-West infrastructure are delivering without any delays.

We will provide an update at 20:00 UTC, or as soon as more information becomes available.

As of approximately 18:00 UTC, we have applied mitigation to restore the service. Queued emails have started getting delivered but at a slow rate.

We will provide further updates as more information becomes available.

We have identified a possible cause of the issue and we are now focused on implementing a solution.

We will provide an update at 19:00 UTC or as soon as more information becomes available.

We are continuing to investigate this issue with high priority. We will deliver queued email following our retry schedule once service is restored.

We will provide an update at 19:00 UTC or as soon as more information becomes available.

We are aware of customers provisioned on NAM based infrastructure experiencing delays sending or receiving emails. We will deliver queued email following our retry schedule once service is restored.

We are handling this issue as a high priority and have engaged all appropriate teams at Broadcom.

Please note that there is no impact to protection services, and all other aspects of our services remain functional.

We will provide an update at 18:30 UTC or as soon as more information becomes available.