Notice #42114
Identified

Degraded Performance on Web App

Reinstating the status of this incident to Identified as high CPU spikes continue to reoccur. We'll continue to investigate.

High CPU usage was identified on some of the web app instances. We've restarted these instances as a first mitigation step. CPU usage appears to be returning to normal but we're still monitoring and investigating the cause of this spike in CPU usage.

We're seeing degraded performance across the web app effecting report generation and organisation syncing. An investigation is underway and we'll keep you updated via this status page.

  • Calxa Online Web App

    Where you sign in and get things done.

  • Integration Services

    Syncing or updating your accounting system.

  • Automation and Workflows

    Processes that run even when your not in the app.

  • Help Centre & Support

    Where you find online help material and get support.

Previous Incidents

[Resolved] Problems signing in to Calxa

This incident lasted 2 days, 5 hours, and 26 minutes.
  • Calxa Online Web App
Mon, 4 Nov 2019
08:21:20 AEST

We have had some users having trouble signing in to Calxa. We are investigating the cause but the workaround is to either clear your browser cache or use another browser such as Edge, Firefox or Safari.

To clear the cache in Chrome, click the 3 dots by your name at the top right, then Settings. Search for cache and it will return Clear Browsing data. Select that and then Clear Data on the next screen.

We will provide more updates as we have more information.

10:05:39 AEST

We've identified the issue that is causing sign in problems. As a temporary fix we've rolled back our latest feature release, which looks to be the root cause of the problem. The new Actual v Budget Balance Sheet report will be available again shortly.

18:26:18 AEST

We've now fixed this issue and you should now be able to sign in without problems. We have also re-published the new Actual vs Budget Balance Sheet report.
Thanks for your patience while we resolved this issue.

Mick Devine

[Resolved] Xero API Down

This incident lasted 18 hours and 4 minutes.
  • Integration Services
Mon, 28 Oct 2019
15:30:33 AEST

Xero API is currently down, as reported on their Status Page. This will be effecting Calxa users trying to Sync with Xero organisations.

https://status.developer.xero.com/

15:52:20 AEST

Xero have updated the incident status as Monitoring with the following update.

"We're currently monitoring the solution we have in place. API is working again although with degraded performance"

16:52:37 AEST

An update has been posted by Xero advising the following.

"We're still working with a technology vendor to resolve the problem of the degraded performance."

Tue, 29 Oct 2019
09:01:33 AEST

The incident was resolved by Xero. Thanks for your patience.

[Resolved] www.calxa.com website is down - access Calxa via https://cloud.calxa.com

This incident lasted 18 hours and 16 minutes.
  • Calxa Online Web App
Sat, 26 Oct 2019
11:49:24 AEST

Our website, www.calxa.com is currently disrupted due to problems with our hosting provider, Siteground.
If you normally go the website to sign in to the app, use the https://cloud.calxa.com address to sign in instead. That takse you straight to the sign-in page.

We will provide updates as the issue is resolved.

14:16:34 AEST

Our website is now back up and running. Because of a DDOS attack on one of Siteground's servers, they disabled access to all sites on that server. We've now moved to another server temporarily while they fix their issues.

We apologise for the inconvenience.

[Resolved] Xero API Not Responding

This incident lasted 48 minutes.
  • Integration Services
Mon, 30 Sep 2019
11:07:32 AEST

We are investigating reports of problems with the Xero API and updating into Calxa.
We will provide updates here once we hear back from Xero

11:32:10 AEST

The Xero API issue appears to have been intermittent and all customers we are aware of with issues are now resolved. We will continue to monitor and discuss with Xero support

11:55:49 AEST

There was a temporary issue with Xero's servers which has now been resolved. Thanks for your patience while we worked through this

[Resolved] Xero API is Currently Down

This incident lasted 20 minutes.
  • Integration Services
Thu, 15 Aug 2019
09:35:11 AEST

Xero API is currently down, as reported on their Status Page. This will be effecting Calxa users trying to Sync with Xero organisations.

https://status.developer.xero.com/

09:55:22 AEST

Xero have rolled back an update to their API and this incident is now resolved.

[Resolved] MYOB Sync or Import Errors

This incident lasted 3 days, 1 hour, and 34 minutes.
  • Integration Services
Mon, 8 Jul 2019
11:16:46 AEST

We've identified an issue where requests made to the MYOB API are intermittently returning server errors. This has been reported to MYOB and they've escalated our reports to their engineering team.

16:55:17 AEST

We're still awaiting an official reply from the MYOB engineering team. In the meantime we've implemented some retry logic that does appear to have improved the completion rate on synchronisation tasks.

Tue, 9 Jul 2019
17:18:06 AEST

The server errors being returned by the MYOB API are still occurring and we're still waiting on further correspondence from MYOB. In the meantime we've reached out to other MYOB add-on developers and confirmed they're also experiencing similar issues.

Wed, 10 Jul 2019
08:56:55 AEST

MYOB have reached out to us this morning and advised that the "Engineering teams are still looking into this issue. Pin pointing a cause is proving more complex than first thought."

11:54:17 AEST

MYOB have made some changes at their end. We will both be monitoring for the next 24 hours to confirm if the issue has been resolved.

Thu, 11 Jul 2019
09:51:24 AEST

We've not seen an error response from the MYOB API now for approx. 18 hours. We'll continue to monitor but the changes made by MYOB yesterday afternoon appear to have resolved their issues.

If you're still experiencing issues with MYOB Sync please contact Calxa support.

12:51:27 AEST

No further errors have been observed and MYOB syncs are completing successfully. We're now closing this incident.

[Resolved] Connection Timed Out for some New Zealand Customers

This incident lasted 23 hours and 44 minutes.
  • Calxa Online Web App
Wed, 5 Jun 2019
14:36:37 AEST

A small number of customers have reported an error connecting to the cloud app. The error shown in the browser is ERR_CONNECTION_TIMED_OUT, but the issue appears to only effect New Zealand customers with Vodafone as their internet service provider.

I'd encourage anyone with similar issues to please report them to helpme@calxa.com as we're seeking more details about this problem.

Thu, 6 Jun 2019
14:21:32 AEST

We've now resolved the incident. Thanks for your patience.

[Resolved] Internal Server Errors

This incident lasted 19 hours and 59 minutes.
  • Integration Services
  • Calxa Online Web App
  • Automation and Workflows
Thu, 9 May 2019
14:22:53 AEST

Internal server errors are causing issues with report generation and organisation syncing. We've started investigating this issue and will keep you updated via this status page.

15:00:49 AEST

Senior engineers are now investigating the problem.

15:34:03 AEST

We've applied a temporary workaround to the internal networking issue. This should make maker things run more smoothly while we work on a permanent solution.

Fri, 10 May 2019
08:58:44 AEST

The workaround we've had in place to access our internal Azure services with a different URL has resolved problems for now.

The underlying issue has been identified and documented as intermittent issues with our DNS provider. You can track that incident here.
https://status.netregistry.com.au/incidents/70rqf0d55gjx

When we feel comfortable that the underlying DNS issue has been resolved we'll remove the workaround and return to our normal URL.

10:21:58 AEST

Our DNS provider has marked the case as resolved and the app has remained stable. If you continue to experience any issues please contact our support team.

[Resolved] Calxa Online Unavailable

This incident lasted 1 hour and 36 minutes.
  • Automation and Workflows
  • Calxa Online Web App
  • Integration Services
Fri, 3 May 2019
06:30:54 AEST

When users open Calxa in the browser a Runtime Error is shown and users cannot login. This problem also effects users accessing via the desktop client. We're now investigating the problem.

07:12:03 AEST

Our services are hosted on Microsoft Azure and they've acknowledged a global DNS issue which is effecting network connectivity. You can monitor the status on the Microsoft outage here https://azure.microsoft.com/en-us/status/.

07:14:51 AEST

We've fixed the core issue, and are waiting for things to recover.

07:17:36 AEST

Most users are now able to sign in to Calxa Online. The Microsoft incident is still active so we will continue to monitor.

08:07:50 AEST

Connectivity has continued to be stable and Microsoft have updated their incident advising of the recovery so we'll mark this as resolved. Please advise if you still experience issues.

[Resolved] Degraded Performance on MYOB Integration

This incident lasted 2 days, 9 hours, and 20 minutes.
  • Integration Services
Tue, 26 Mar 2019
19:30:44 AEST

We're currently investigating an issue that is causing MYOB API requests to time out. Some users are unable to sync Calxa from MYOB due to these request timeout errors.

Wed, 27 Mar 2019
05:34:18 AEST

Overnight we've continued investigating the timeout issues with MYOB API requests and we feel there has been some degradation in performance from the MYOB API side. We've reached out to MYOB for their thoughts on this but as a workaround we've added retry logic after a timeout failure. This way the syncs don't just fail due to the timed out requests, however the syncs are still taking longer than usual.

16:54:18 AEST

Still investigating the slow response time on MYOB integration. We're now working with MYOB developer support to troubleshoot.

The workaround applied this morning has solved the issue of failed syncing but the process is still running slow.

Thu, 28 Mar 2019
09:13:58 AEST

The response times seem to be improving from the MYOB side and based on some feedback received from their support we're also working on improved request logic. We'll be rolling this out over the course of today and will continue to monitor.

Fri, 29 Mar 2019
04:51:15 AEST

MYOB API requests are no longer timing out. In addition we've made significant improvements to our own syncing logic such that organisations with large numbers of job transactions are now syncing much faster.

[Resolved] Xero Accounting API currently unavailable

This incident lasted 29 minutes.
  • Integration Services
Wed, 30 Jan 2019
10:36:51 AEST

Xero have reported an incident with their Accounting API that is causing Calxa updates from Xero organisations to fail. More details can be found on the Xero Status page.

https://status.developer.xero.com/incidents/jwl58vcrp0w5

10:55:50 AEST

Xero have now rolled back an earlier update and the API endpoints are responding. Calxa integration to Xero is now operational.

11:06:41 AEST

We've now conducted internal testing and confirmed the Xero integration is fully operational. If you're still experiencing issues updating from Xero please contact Calxa support for assistance.

[Resolved] Web App returning internal errors - Users see bubbles or server errors

This incident lasted 3 hours and 2 minutes.
  • Calxa Online Web App
  • Integration Services
  • Automation and Workflows
Fri, 11 Jan 2019
09:17:21 AEST

Investigation is underway. We'll continue to update this status page as more is known.

09:39:54 AEST

Developers have identified the cause and working towards a fix. Still unable to advise an estimated time of resolution.

10:42:27 AEST

We've gathered additional information regards this problem which has occurred due to an update on our hosted servers. Unfortunately the fix requires development and will take some more time to resolve.

11:12:40 AEST

A fix is now being applied. Services should start to return to normal. We'll continue to monitor and test all services. We'll provide a final update when all testing has been concluded.

12:19:40 AEST

We've now resolved the incident. Thanks for your patience.

No further notices from the past 365 days.