Service Recovery Information
Status: all Corvirtus solutions are currently online and operating. Our team welcomes your questions and feedback.
Root Cause Analysis Summary - June 3, 2023 Outage
Thank you for your patience as our technical team and leadership worked closely with our managed services provider to provide this root cause summary. We appreciate the stress and delays the outage caused your team and are working to ensure it won't happen again.
What happened?
On June 3rd, 2023, our managed services provider published a security and patch update to our software but made a mistake. This script included a cleanup routine intended to remove temporary files and free up disk space, but contained a coding error that corrupted the Windows operating system of our server. This required reinstallation and restoration from our backups taking several days. While we had a redundant server in place, the update was pushed to both at once causing the outage.
How did we respond?
Our technical team was alerted immediately of the outage on June 3rd and followed our disaster recovery protocols and alerted Corvirtus leadership. Our managed services provider was already engaged as this error affected most of the customers. We made progress quickly and the servers were operational again on the 6th.
What caused the problem?
There was a coding error in the update initiated by our managed services provider. All steps for testing and quality assurance were not followed.
How are we preventing this in the future?
We're changing our process to make sure this doesn't happen again. We'll have better controls and tests to catch any problems. We're in close communication with our managed services provider to ensure the correct procedures are followed for every update and our technical team is consulted. The technician who made the error no longer has access to Corvirtus systems. Going forward, all updates to our technology will use automated testing that ensures all updates deployed are thoroughly tested and approved. We are also ensuring our redundant servers are not updated at the same time as our primary servers so if an error occurs we are still fully operational.
Our goal moving forward is to minimize potential future disruptions by enhancing our systems and ensuring our partners are capable of delivering high-quality solutions, increasing efficiency, and doing whatever it takes to meet and exceed your expectations. We value your trust in us and we are committed to your success.
For a detailed root cause analysis report you are welcome to contact us at info@corvirtus.com.