Email Delivery Issue in EMEA Region

Resolved
Resolved

The previous reported concern with email delivery in the EMEA region has been resolved. The service is fully restored and there is no longer an impact to customers. This incident is resolved.

Update

The email service remains fully operational and we continue to closely monitor the email traffic.

We will provide our next update on May 1, 2020, at 20:00 UTC, or as new information is discovered.

Monitoring

EMEA-Central Infrastructure The EMEA-Central infrastructure continues to process emails without any issue.

EMEA-West Infrastructure At 14:25 UTC, the EMEA-West infrastructure migration to the Broadcom network backbone was successfully completed. As of 14:53 UTC, the site has been brought online and is processing email traffic without any delays.

The email service is fully operational and we continue to closely monitor customer traffic.

We will provide an update as soon as more information becomes available.

Update

EMEA-Central Infrastructure At 12:30 UTC, the faulty optics cable has been replaced and as of 13:20 UTC, the EMEA-Central infrastructure has been placed back in service. All new emails are now being processed by EMEA-Central site. The previously queued emails continue to deliver based on retry schedules.

EMEA-West Infrastructure As of 14:20 UTC, the EMEA-West site has been taken out of service to perform the migration to Broadcom network backbone.

Any previously queued emails on sending servers (MTAs) are now being processed by EMEA-Central site and a vast majority have been successfully delivered

We will provide an update as soon as more information becomes available.

Identified

EMEA-Central Infrastructure The engineers have discovered and replaced a faulty optics cable onsite and we are currently performing tests to ensure the underlying issue has been resolved before restoring it to service. Queued emails in the EMEA-Central infrastructure will be delivered once the site is back online.

EMEA-West Infrastructure The EMEA-West site remains up and processing email, customers may see some backlogs on their MTAs which accrued during the connectivity loss. The backlogs are processing and should clear soon.

We will provide an update as soon as more information becomes available.

Investigating

Engineers have arrived onsite and are in the process of confirming that the hardware is fully functional. Emails queued on the EMEA-Central infrastructure will be delivered once the site is back online.

The EMEA-West site remains up and processing email. We will provide an update as soon as more information becomes available.

Update

We continue to investigate the underlying issue impacting the EMEA-Central site. Emails queued on the EMEA-Central infrastructure will be delivered once the site is back online.

The EMEA-West site remains up and processing email. We will provide an update as soon as more information becomes available.

Update

We continue to investigate the underlying issue on the EMEA-Central site. Emails queued on the EMEA-Central infrastructure will be delivered once the site is back online.

The EMEA-West site remains up and processing email, customers may see some backlogs on their MTAs which accrued during the connectivity loss. The backlogs are processing and should clear soon.

We will provide an update as soon as more information becomes available.

Update

As of 04:18 UTC, the EMEA West site is operational and continues to process all EMEA traffic. Emails queued on the sending servers should start delivering.

The EMEA Central site remains out of service and troubleshooting continues to bring the site back online. Any emails queued on EMEA Central infrastructure will be delivered once the site is back online.

We will provide an update as soon as more information becomes available.

Identified

We have implemented a work around, customers connecting to the EMEA infrastructure should now be able to send and receive email. Our engineering team is currently working to remediate the underlying cause for this issue.

We will provide an update at 06:00 UTC or as soon as more information becomes available.

Update

Initial investigation points to a network connectivity issue with EMEA sites. Our engineering team is investigating this issue as high priority to determine the underlying cause. Customers connecting to the EMEA infrastructure may continue to experience connectivity issues.

We will provide an update at 05:30 UTC or as soon as more information becomes available.

Investigating

We are aware of customers provisioned on the EMEA based infrastructure experiencing issues sending and receiving emails. We are investigating this issue as a high priority and have engaged all appropriate teams at Broadcom.

We will provide an update as soon as more information becomes available.

Began at:

Affected components