All Nodes - Enterprise Monitoring Point loss of Experience and Delivery monitoring data

Resolved
Resolved

AppNeta Engineering has applied software fixes in last night's deployment to address the issues referenced in this status page. At this point, Monitoring Points are connecting back , have the proper statuses applied in APM, and Monitoring Point upgrades are re-enabled. We consider this issue resolved and will post the results of our postmortem.

Update

AppNeta Engineering has taken steps to address the sequencers stuck in an upgrading state and has confirmed that the telemetry is flowing for majority of these these as well. We will leave Monitoring Point upgrades disabled through the weekend and as we produce long-term fixes for the issues that have lead to this.

We will continue to monitor this issue going forward.

Update

Maintenance against App-03 and App-13 application nodes is now complete.

Update

AppNeta Engineering is going to perform emergency maintenance against the App-03 and App-13 application nodes, in relation to this issue. During this brief time (1-2 minutes), APM will be unavailable. This will take place at 6:15pm (EST)

Update

In an effort to avoid more instances of failed Monitoring Point upgrades, AppNeta Engineering is disabling the ability to upgrade Monitoring Points on demand.

Update

While Monitoring Point connections and telemetry have remained stable through the night, we have identified an issue with some Monitoring Points, which has caused them to stall in their upgrade. During this time, those monitoring points will be inactive, not generating telemetry or running diagnostics. AppNeta Engineering is working on this issue and will post updates here.

Monitoring

Emergency maintenance is complete and we can see monitoring data coming in on application nodes. We will continue to monitor this issue.

Update

AppNeta Engineering will be performing emergency maintenance across all application nodes, to address this issue. APM will be briefly interrupted for a short period of time across the next 30 minutes.

Investigating

AppNeta is experiencing a loss of Experience and Delivery monitoring data from Enterprise Monitoring Points. AppNeta Engineering is investigating the issue.

We will provide updates as we work through the issue.

Began at: