Network connectivity issue in NAM region

The previous reported concern with email delivery in the NAM region has been resolved. The service is fully restored and there is no longer an impact to customers. This incident is resolved.

The service remains fully operational and customers should continue to see mail flowing normally.

We will continue to closely monitor the email service and will provide updates as new information becomes available.

Previously queued email on the US based infrastructure have now been cleared with email now processing as expected.

We will continue to closely monitor the service and will provide our next update at 13:00 UTC, or as soon as more information becomes available

Customers provisioned on the US based infrastructure continue to experience delays sending or receiving emails. The configuration changes on the US East infrastructure is clearing backlog but at a slow rate. We will be closely monitoring the changes completed on US East before applying the change to the US West infrastructure.

We will provide our next update at 11:00 UTC, or as soon as more information becomes available.

Customers provisioned on the US based infrastructure continue to experience delays sending or receiving emails. This is related to the load on US infrastructure due to the previously queued emails. We have made a configuration change on one backend MTA and we are monitoring the results before implementing to additional MTA's.

Email queues continue to clear but at a slow rate.

We will provide our next update at 08:00 UTC, or as soon as more information becomes available

Customers provisioned on the US based infrastructure continue to experience delays sending or receiving emails. This is related to the load on US infrastructure due to the previously queued emails. Our operations team is reducing the load by taking subset of infrastructure out of service in rotations to allow emails to drain. Email queues continue to clear but at a slow rate.

We will provide our next update at 06:00 UTC, or as soon as more information becomes available

Update - Customers provisioned on the US based infrastructure continue to experience delays sending or receiving emails. This is related to the load on US infrastructure due to the previously queued emails. Our operations team is reducing the load by taking subset of infrastructure out of service in rotations to allow emails to drain. Email queues continue to clear but at a slow rate.

We will provide our next update at 04:00 UTC, or as soon as more information becomes available

Customers provisioned on the US based infrastructure continue to experience delays sending or receiving emails. This is related to the load on US infrastructure due to the previously queued emails. Our operations team is reducing the load by taking subset of infrastructure out of service in rotations to allow emails to drain. Email queues continue to clear but at a slow rate.

We will provide our next update at 02:00 UTC, or as soon as more information becomes available.

We are aware of a subset of customers provisioned on the US based infrastructure experiencing new round of delays sending or receiving emails. This is related to the load on US infrastructure due to the previously queued emails. Our operations team is reducing the load by taking subset of infrastructure out of service in rotations to allow emails to drain. Email queues continue to clear but at a slow rate.

We will provide our next update at 23:30 UTC, or as soon as more information becomes available.

Email Delivery Issue to Google Domains Previously queued email on Broadcom infrastructure continues to deliver but at a slow rate. Queues are trending downward and we are closely monitoring the service.

Email Delays in NAM Region Backlog generated on Broadcom infrastructure during this issue is clearing but at a slow rate. We are looking into options to increase the rate of delivery. Any remaining emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

Track and Trace Issue Track and Trace service is mostly operational. Some customers may continue to experience delays for emails passed through US East infrastructure while the remaining backlog clears.

We will continue to closely monitor the email service and will provide our next update at 23:30 UTC, or as soon as more information becomes available.

Email Delivery Issue to Google Domains Previously queued email on Broadcom infrastructure continues to deliver. Queues are trending downward and we are closely monitoring the service.

Email Delays in NAM Region Backlog generated on Broadcom infrastructure during this issue is clearing. Any remaining emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

Track and Trace Issue Track and Trace servers are back in service and processing searches. Some customers may continue to experience delays for emails passed through US East infrastructure while the backlog clears.

We will continue to closely monitor the email service and will provide our next update at 19:30 UTC, or as soon as more information becomes available.

Email Delivery Issue to Google Domains At 14:50 UTC, a change was made by Google to mitigate the issue. This has resulted in the restoration of connectivity issues previously experienced. We have placed the US West infrastructure back in service and previously queued emails have started delivering.

Email Delays in NAM Region We have addressed the underlying issue in the US East infrastructure. Customers should see immediate improvement with new emails. Backlog generated during this issue has started clearing gradually.

Track and Trace Issue Track and Trace servers are back in service and processing searches. Some customers may continue to experience delays for emails passed through US East infrastructure while the backlog clears.

We will continue to closely monitor the email service and will provide our next update at 17:30 UTC, or as soon as more information becomes available.

We are aware of customers provisioned on the NAM based infrastructure experiencing delays sending or receiving emails. In addition to email delays, a subset of customers may have received errors when connecting to the impacted infrastructure resulting in emails queuing on customer and/or third party MTAs.

Emails previously sent to Google domains during the DNS issue will remain queued on our infrastructure and will be delivered once the impacted infrastructure has been put back into service.

We are currently investigating this issue and will provide the next update at 16:30 UTC or as new information becomes available.

We continue to work on restoring the impacted infrastructure. Emails sent to Google domains during the DNS issue will remain queued on our infrastructure and will be delivered once it is has been put back into service. All new emails are being delivered as expected.

We will continue to closely monitor the email service and will provide our next update at 20:00 UTC, or as soon as more information becomes available.

As we started putting the previously impacted infrastructure back into service, we noticed that emails sent to a Google domain were not being delivered due to a DNS issue. The impacted infrastructure was taken back out of service with new emails sent to a Google domain are now being delivered as expected. Emails to a Google domain sent during the impact window are queued on our infrastructure. They will be delivered once the issue has been resolved and it is put into service.

We will continue to closely monitor the email service and will provide our next update at 20:00 UTC, or as soon as more information becomes available.

Between approximately 02:45 and 3.15 UTC, customers provisioned on the Email Security.cloud NAM infrastructure, may have experienced intermittent connectivity issues for mails going through one of the two regions. Mail traffic is balanced between the two regions, US-West & US-East. At 3.15 UTC the impacted infrastructure was taken out of service to mitigate the issue, both regions are operating as expected.

We will continue to closely monitor the email service and will provide our next update at 20:00 UTC, or as soon as more information becomes available.

Connectivity has been restored for customers provisioned on our Email Security.cloud NAM infrastructure. Customers should see emails flowing normally.

We will continue to closely monitor the email service and will provide our next update at 08:00 UTC, or as soon as more information becomes available.

Please be advised that we are currently experiencing network connectivity issue to the US-West 2 Region. Our team is working on this issue at highest priority. Please note that there is no customer impact as we have switched over to a parallel site for business continuity. We are currently investigating this issue and will provide updates as new information becomes available.