Degraded Performance on Web App

Conclusion: Degraded Performance (15th Nov)

After further investigation we found poor closing balance calculation logic that meant values could in some cases be recalculated many times for a single report. For multi-period reports where there are many periods or when consolidating multiple entities this exponentially heightened the problem.

We have now improved the closing balance calculation logic so that calculated values are cached and reused when required, instead of recalculated. We have also improved the online hardware and scale configurations, then stress tested these new configurations, so we're confident that a similar incident will not happen again.

We've now resolved the incident. The cause of CPU spikes was related to some specific report calculations and we've mad some improvements to our calculation logic to reduce the impact on CPU. Thanks for your patience.

Reinstating the status of this incident to Identified as high CPU spikes continue to reoccur. We'll continue to investigate.

High CPU usage was identified on some of the web app instances. We've restarted these instances as a first mitigation step. CPU usage appears to be returning to normal but we're still monitoring and investigating the cause of this spike in CPU usage.

We're seeing degraded performance across the web app effecting report generation and organisation syncing. An investigation is underway and we'll keep you updated via this status page.