Mike Boylan

I want to tell you a very spooky 👻 story of the time prod drowned in a hurricane. The year was 2012. I was the Director of Engineering at a startup in Durham, NC. Our entire infrastructure was hosted on bare metal with Internap in Manhattan. And then Hurricane Sandy made landfall. Internap assured its customers that the servers were on the 26th floor and therefore completely safe. They were not safe. For years I had been unhappy with the speed of hardware upgradability. When I wanted to bump up the memory for MySQL I had to print out a form, sign it, fax it in, and wait for an engineer to take a train to the server building and upgrade the boxes. I decided that Sandy was the perfect time to experiment. JUST IN CASE. A couple days before Sandy was projected to make landfall, I set up a MySQL replication target in AWS us-east-1 and made sure everything was syncing correctly. I already had a cron job syncing over our file repository from NFS to an S3 backup every few days and I changed that to be hourly. JUST IN CASE. At this point I was betting on a disaster scenario so I was thinking of all the worst things that could happen. It’s important to note that: 1) I was the sole decision maker so there was little to no second guessing of my decisions 2) No one was telling me to stop freaking out over nothing The morning of Oct 29, the news indicated that Sandy would be making landfall that night. Our db server had a replication backup to the millisecond. Our file servers had a backup to the hour. Our memcache servers were ephemeral. Our jobs were running on a little beanstalkd server. The web servers were running on a new shiny product at Internap — VIRTUAL MACHINES — so I didn’t need to worry about them at all. Three days ago Internap had emailed saying that in the event of a power outage they had enough diesel on site for 65 hours of generator power. So I really just needed to calm the f down instead of all this disaster planning. --- Email from Internap at 7:21pm: UPDATE: The LGA11 facility reports that commercial utility power is no longer available and the site has successfully transferred to generator. Site engineers have completed a walk-through and all systems appear to be functioning as designed. --- Well, I'm glad everything is under control and I have NOTHING to worry about. At least you have 65 hours of fuel reserves for me to get my shit together. --- Email from Internap at 11:01pm: [...] flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. [...] available reserves on the mezzanine are estimated to support customer loads for approximately 5-7 hours. the customer support team will begin shutting down your servers gracefully at 12:30am EST to avoid damage to your equipment. For our cloud customers, we will also begin shutting down the infrastructure at this time. --- WAIT, WHAT. The "cloud" is just someone else's flooded basement. So I hurriedly login to my AWS account, spin up a couple EC2 instances, and run my Vagrant scripts against them to set them up from scratch. As this are happening, the database server back in Manhattan completely dies. Drowns? RIP my inbox oh fuck, dns I switch dns to point to the load balancer for the two EC2 instances that aren't even up yet because Ansible is still like installing nginx or something. The TTL was already like 15 seconds because come on, I'm not a complete muppet. Everything is down. All the alerts are screaming. My coworker Sarah messaged me at 1204am (26 minutes before the time Internap said they would shut everything down) with "server issue?" This was such a welcome moment of comic relief for me. I kinda laughed, kinda cried. The EC2 machines were finally up! The application code (HURRAY MONOLITH) was checked out and ready because thankfully Github was not running on bare metal in New York. I have never been so glad as I was then to not be running a microservice architecture or some other distributed system. DNS started to resolve and...nothing worked. Oh! First things first, I promoted the MySQL server. Next the config variable for the MySQL host was all wrong. Fixed that in config and built the app again. Chat logs show that I told Sarah that the site was working again at 1208am. We had a total of 17 minutes of complete downtime. Of course file downloads of deliverables from S3 didn't work because the code was still pointing to the old NFS server, and emails weren't going out because I forgot about the jobs server, and db queries were slow af coz I didn't bother setting up memcache. First I set up beanstalkd. Have you met my friend beanstalkd? Such a little beauty. So I installed it and emails were working again. Then set up memcache, and boom, mysql is fast again. Ensure people could still buy things. Logins and sessions all worked. My brain had effectively stopped working at this point. I'd been working nonstop for 18 or so hours. This was a period of time when I was the Director of Engineering, the head of infra, also wrote code, and the org chart showed me reporting to myself. Unfortunately I only got one salary. It was a very dark period. Highly recommend against being an EM+ who codes. I emailed the entire company giving them a quick debrief of what was happening, and finally went to sleep. When I woke up the next day, we fixed all the other weird little things that were broken. And that is the story of the most ridiculous on-prem to cloud migration.