All Application Nodes - Violation Triggered Diagnostics - Failing to Execute

Broadcom Engineering has continued to monitor this issue through the day, following the patch deployment last night. We have seen no new instances of the identified problem and now consider this issue resolved.

Broadcom Engineering completed the planned patching last night, and all diagnostics appear to be executing as expected at this point. We will continue to monitor the issue today and will post any further updates here.

Broadcom Engineering has identified the source of this issue as an application defect. We are planning to address this issue via software patch, to be deployed on Wednesday, June 21st, at 8pm EDT. See this maintenance status for details.

Broadcom Engineering is continuing our investigation, but has a better understanding of the scope and nature of this issue. This problem is affecting network paths that are violating with a frequency of less than thirty minutes. In such cases, upon the initial violation, the diagnostic will execute as expected, but subsequent violations will fail to execute the expected diagnostic. Subsequent diagnostics will fail to execute until greater than thirty minutes has elapsed between violations of the affected network path.

Broadcom Engineering has identified an issue affecting all application nodes, which is causing diagnostics triggered by network path violations to fail to execute. This appears to be affecting only paths that are violating very frequently. Manually triggered diagnostics are not affected.

Broadcom Engineering is working to clarify the scope and cause of the issue, and will post updates here.