Resolved

As of June 3rd 2018, we have not seen a re-occurrence of the delivery issues to Microsoft Office365. We have confirmed service is fully restored. The incident is now resolved.

Update

We are no longer seeing emails experiencing throttling and mail queues have now cleared. Our internal tests so far confirm that rate-limiting from Microsoft is no longer present. Our engineers are in the process of introducing a small number of servers back in service.

We will continue to closely monitor email traffic for the next 24 hours before marking this fully restored.

The next update will be provided on June 4th at 16:00 UTC, or once we have new information available.

Update

At 04:50 UTC, Microsoft removed throttling for the impacted IP range, and email delivery was expected to return to normal. Our tests later confirmed that new and existing emails were still getting queued. As a result, a decision was made to take the impacted servers out of service.

Our engineering team is continuing to provide email samples and error logs to Microsoft and using various sources to have this rate-limiting removed. We request our customers whitelist the IP ranges and report delivery issues directly to Microsoft.

The next update will be provided on June 3rd at 20:00 UTC, or once we have new information available from Microsoft.

Update

We are continuing to work closely with Microsoft to address the delays. We will provide further updates as new information is discovered.

We are requesting affected customers to open a case directly with Microsoft and request rate-limiting to be removed as well as the IP range to be added in global whitelist. Customers are also encouraged to whitelist Symantec IPs using Microsoft Office 365 solution.

Monitoring

We are currently experiencing issues delivering emails to a small subset of customers/recipients using Microsoft Office365 service. The result is that emails are queued for delivery. Our investigation shows the issue is occurring due to Microsoft throttling our 67.219.240.0/20 IP range. As a workaround, our engineers have removed the affected IP range from production to avoid service interruption. However, any existing email will remain queued till the throttling is cleared.

We are working with Microsoft to address the situation. We are requesting affected customers to open a case directly with Microsoft and request rate-limiting to be removed as well as the IP range to be added in global whitelist. Customers are also requested to whitelist Symantec IPs using Microsoft Office 365 solution.

Symantec will continue to hold emails based on standard and custom retry email schedules configured by our customers.

We will provide further updates once we have more information from Microsoft.

Began at:

Affected components