Beijing (PEK1) Data Center Unavailable

Resolved
Resolved
After 1 month, 1 week, 13 hours, and 51 minutes

We have monitored the service for stability and this incident shall now be closed.

Monitoring
After 1 month, 4 days, 21 hours, and 11 minutes

We are happy to announce that the issue is resolved and access to Beijing (PEK1) site has been restored.

We will now monitor the service for stability and provide updates as soon as new information becomes available.

Identified
After 6 days, 20 hours, and 32 minutes

Important Beijing (PEK1) Availability Update

We apologize for the extended outage in the Beijing (PEK1) data center, which was caused by a government regulatory intervention beyond our control. Please be assured that Symantec remains committed to providing world-class service in mainland China. The vast majority of customer traffic has already failed over to the Shanghai (SHA1) data center, eliminating or minimizing impact to the user experience. The goal of this communication is to ensure that all remaining traffic in Beijing is moved to Shanghai.

In order to comply with Chinese regulations we’re required to temporarily cease transiting all traffic from the Beijing data center. Therefore, if you explicitly direct traffic to Beijing, such as with an IPsec VPN tunnel, you must move your connection to the Shanghai (SHA1) data center before June 3, 2022, 00:00 UTC to avoid loss of service. All WSS Agent connections and traffic directed to proxy.threatpulse.net have already been automatically rerouted to Shanghai by Symantec.

The Shanghai (SHA1) data center continues to operate normally and we have verified that the issue impacting Beijing does not affect the Shanghai site, nor do we expect any impact in the future. We will continue to monitor Shanghai closely to ensure a good user experience. Even so, our goal is to restore Beijing as quickly as possible to provide full POP redundancy.

Presently, we estimate that the Beijing site will remain unavailable for at least 60 days while we continue work with local authorities to resolve the issue. However, complex factors, including COVID lockdowns, are preventing a timely resolution. We will continue to work on the problem with urgency until it is fully resolved and will provide updates as they become available. Visit https://status.broadcom.com to subscribe to service availability push notifications and updates.

Shanghai (SHA1) Ingress IP Address:

  • IPSec, trans-proxy: 222.126.180.164

Shanghai (SHA1) Egress IP Addresses:

  • 222.126.180.0/24
  • 222.126.181.0/24

Required Action

  • Move all remaining IPsec and explicitly directed connections from the Beijing (PEK1) data center to the Shanghai (SHA1) data center before June 3, 2022, 00:00 UTC.
  • If not already done, update any firewall rules regulating connectivity with WSS to allow traffic to pass to/from the Shanghai (SHA1) IP networks listed above.
  • Third party applications that regulate connections by source IP address should be updated to accept connections from the Shanghai (SHA1) IP networks listed above to ensure traffic proxied through WSS can reach the applications.

Failure to make these changes could result in loss of service, users unable to connect to WSS, issues accessing third party web applications, or issues authenticating against the service using the Auth Connector (where applicable).

  • WSS Agent, SEP Agent, and Unified Agent: Agent traffic will be automatically redirected to Shanghai (SHA1). No action is required.
  • Explicit proxy and proxy forwarding: Customers directing traffic to proxy.threatpulse.net will be automatically redirected to SHA1. No action is required.

Please visit this KB article for a full list of IP networks used by WSS:

https://knowledge.broadcom.com/external/article?legacyId=TECH242979

Technical Support

Experiencing issues? Contact technical support by visiting: https://support.broadcom.com/security.

For service status and maintenance updates visit and subscribe to Broadcom Service Status: https://status.broadcom.com.

Identified
After 3 days, 20 hours, and 33 minutes

We are continuing to engage with the local carrier to remove the block on the IP addresses for the Beijing (PEK1) data center. We will provide an update as soon as further information is available.

Identified
After 17 hours and 59 minutes

We have been advised that the removal request for the Beijing Data Center IP block has been forwarded to the local government authority and will be reviewed and processed sometime early next week.

For all connection methods except IPsec, we have failed over customer traffic to Shanghai (SHA1) Data Center. We have a workaround available for IPsec and request customers to get in touch with our support team to discuss it further.

We will provide updates as soon as new information becomes available.

Investigating
After 9 hours and 34 minutes

We are continuing to engage with the local carrier to remove the block on the IP addresses for the Beijing (PEK1) data center. We will provide an update as soon as further information is available.

As a reminder, customers connecting to the Beijing (PEK1) data center may continue to experience service disruption.

Investigating
After 6 hours and 50 minutes

We continue to wait for the local carrier to address this issue. As per our investigation, the IPs for Beijing (PEK1) data center are currently blocked. Customers connecting to the Beijing (PEK1) data center may continue to experience service disruption.

We will provide updates as soon as new information becomes available.

Investigating
After 4 hours and 24 minutes

We have escalated this issue to the regional carrier and are waiting for them to investigate this further. Customers connecting to the Beijing (PEK1) data center may continue to experience service disruption.

We will provide updates as soon as new information becomes available.

Investigating
After 1 hour and 56 minutes

The Beijing (PEK1) data center remains unavailable. We are looking into alternative solutions but at this point we do not have anything available. Customers connecting to the Beijing (PEK1) data center may continue to experience service disruption.

We will provide updates as soon as new information becomes available.

Investigating
After 1 hour and 34 minutes

We are continuing to investigate this issue as high priority. As per our preliminary investigation, it is suspected as a Carrier issue. We have opened a priority ticket with regional carrier and waiting for additional updates.

We will provide updates as soon as new information becomes available.

Investigating
After 34 minutes

The Beijing (PEK1) data center is currently unavailable. Existing customers connections may continue to work but any new connections will experience failures.

We are actively investigating and will provide updates as soon as new information becomes available.

Investigating

We are currently aware that some customers connecting to the Beijing (PEK1) data center may experience decreased performance or disconnects from the service.

We are currently investigating and will provide updates as soon as new information becomes available.

Began at:

Affected components