ICANN57: A Real-Life Audio-Video Disaster Recovery Story
Very few people realize what it actually takes to make an ICANN meeting run smoothly from the Network Operations Center backroom, where the InfoTech team orchestrates its magic with high energy. You’ve probably experienced the excellent remote capabilities to encourage diverse participation, good wifi on site, and generally, things run smoothly so you can focus on the real issues and discussions at hand. And this is a good thing.
After Helsinki, we were starting to get prepared again...when we got the news that there was a fire aboard the ship which contained our equipment being transshipped from Helsinki to Hyderabad. (Spoiler alert: I’m pleased to confirm all equipment is now in Hyderabad, and the team is setting up for what will again likely be an excellent meeting.)
Let me share this remarkable disaster recovery story.
When we began preparing for ICANN57 months ago, it was all fairly routine. We packed and shipped all of our technical equipment in a 40-foot and 20-foot sea container which included: 80+ crates containing over five miles of network cables, over 400 microphones, 650 international power strips, 75 MacBooks. We were preparing to get 800 mbps of bandwidth, at least two Internet service providers with redundant paths, and more.
Within a week of the fire, we were told that the 40-foot container was in good shape, but the 20-foot container had possible damage from the fire and would be held for further inspection in Germany.
We were instantly struck by the unique dimensions of this extraordinary incident. A fire on-board a seafaring container cargo ship, while the ship is docked in Germany, en-route to India. We also discovered how complicated the whole situation was given the ship belongs to an international company, the contents belong to a US corporation, the forwarding-agent is based in the US with inspection-rights governed by (200+ year old) maritime laws and with Insurance brokers and adjusters involved from both sides of the Atlantic. The multi-national dimensions of this incident were quite remarkable.
We immediately made back up arrangements. Having catalogued exactly what was in each crate, we enacted two plans: a) a short-term plan to purchase replacement equipment so ICANN57 experiences no disruption; and b) a longer-term plan to work with insurance brokers (under maritime law) to recover the ‘stuck’ 20-foot container. This took a lot of work in a short amount of time, given the sophistication of our gear, which in total cost ICANN about $700,000 USD of unplanned expenses (the equipment itself plus shipping and other costs).
But the real story is that of the ICANN InfoTech team. The manner in which this fire incident was handled is just another example of the professionalism and dedication that the ICANN org team members take to ensure we support the community. I am so proud of our team for working days, nights and weekends to work with every supplier, replace all equipment as well as pack it (our office was a bit of a mess!) and make arrangements to have it air-lifted to India. This was all to ensure that equipment would arrive in Hyderabad in time so that there will be no disruption to our service we provide at ICANN57. Take a look at this time-lapse video from ICANN49 to see just how much goes into setting up an ICANN Meeting.
I wish everyone safe travels if you’re heading to India and hope to see many of you soon – either in person or online.