Resiliency in practice (a story from 2009)
It’s been a bit quiet on our blog recently. The long warm summer had something to do with it, followed by a busy month of September. And then finally, something happened that really shook our world.
On the night of the 1st of October 2009, a significant portion of the Pottelberg site in Kortrijk caught fire. In the deep of the night, Sibren (COO), Julie (Management Assistant) and I (CEO) watched with dismay as the offices of Vivaldi Software were swallowed in flames.
2nd of October 2009. An inferno left nothing of our company. Everything was gone. Everything... except our crown jewels: our (digital) data. This survived because our backup plan and disaster recovery plan includes offsite backups.
The aftermath of the fire was a hectic confrontation with the concept of “resilience”. It would now become clear just how resilient our organization and employees are. Only... this was not a test, this was deadly serious. It was about the survival of our company. Only now do we understand why we discussed those “what-if” scenarios almost every week.
In a 5:00 am crisis meeting in my living room, two managers were designated to take control of the situation. Sibren remained in charge of operational continuity, and I took responsibility for the restart, rebuild and continuity of all supporting services. Shortly after 5:00 am, we already had an announcement on our website reassuring customers that the continued existence, and even the operational continuity, of the company were not in jeopardy. It takes more than a fire to bring Vivaldi Software to its knees.
By afternoon time, we could be reached again by phone. By 2:00 pm, so not even twelve hours after the fire broke out, all internal failover systems in the data center were operational and the developers could get back to work. The SaaS servers which customers use for our Vivaldi software, were safely running in that same data center and did not experience any downtime. Our disaster recovery procedure had succeeded. On Monday, thanks to Lode De Geyter from HOWEST, we had temporary accommodation. We had a central location again where we could work together on the “now” and “the future.” In the weeks, and now months, that followed, we did not have to postpone a single training session, workshop, installation, implementation or even a commercial meeting besides perhaps one small exception. The phrase “the show must go on” could not have expressed it better. This was a huge feather in the hat for all the employees. Pure class! The numerous words of praise we received from our customers for our manner of “tackling” the problem did us good. Thank you, dear customers.
Thank you FDA. Thank you all our customers in aerospace, the medical devices & pharmaceutical sector for being so demanding of us. Thanks to you, we were prepared for this. Thanks to our extensive risk management, the damage remained relatively limited. That resiliency allowed me to write this blog article from our temporary offices in Roeselare. The “Vivaldi Software” phoenix has risen from the ashes stronger than ever.
I can “warmly” recommend a well thought-out disaster recovery plan to everyone. Be thorough!
Don’t worry that you are “paranoid.” Be meticulous, keep it up-to-date, place it on the meeting agenda regularly and hope that you will never need it...