Resolved

After thorough monitoring we are pleased to declare this incident as full restore. In the last 3 hours we have not seen any issues resulting in mail delays. Please accept our apologies for any inconvenience caused. This will be the final update for this incident.

Update

Please be advised that both data-centers in NAM region are now fully operational and servers are processing mail without any issues. Please note that small portion of queued email may still getting drained via your email gateways. Our engineers are closely monitoring the situation and we plan to provide final update at 18:00 UTC once we have analyzed traffic during peak U.S business hours.

Monitoring

At 13:00 UTC our engineers began to return our data centre to service and we expect to return to full capacity around 13:45 UTC. In the mean time please be aware that you may experience some outbound emails queuing on your email gateways. We will make a further posting at 14:00 UTC.

Update

Investigating - Our datacenter remains out of service at this time and all customer email is flowing without delay via alternate locations. Please be aware that our engineers are carrying out tests and some mails in the retry queue may be delivered. We will make a further update to this posting at 15:00 UTC or as soon as we have further information.

Investigating

At 09:50 UTC the infrastructure team took a datacenter serving our US and APJ customers out of service for emergency maintenance. This action will have no impact on our ability to deliver customer emails and no email delays are expected. However the processing of emails which were queued for retry (mails which we have already attempted to deliver and are waiting for another retry attempt) will be paused until we bring the DC back online. We will make a further update to this posting at 12:00 UTC or as soon as we have further information.

Began at:

Affected components