
Builds and operates billion-QPS distributed systems. Senior Director @MongoDB. Core infra, databases, reliability, and engineering management. Views are my own.
Dublin, Ireland
Joined May 2009
- Tweets3.8K
- Following445
- Followers3.4K
- Likes11.5K
A lesson on the value of redundancy for improved uptime: Either your bank should have a multi-zone deployment, or you should have two banks.
🚨 Google lost power in London this morning and Revolut, a bank with 80 million customers, went down with it.
I've spent the morning reading what Revolut users went through. It's wild.
People stranded at petrol stations, unable to pay for fuel or a taxi to the airport.
One traveller couldn't get out of a train station in Japan.
A 71-year-old woman stood at a supermarket till for an hour. Her kid had to drive 30km to bring her cash.
Card payments, transfers, ATM withdrawals, currency exchange, share trading. All down.
Revolut's statement said "all our customers' money and data are safe."
Safe. You just can't touch it.
Google's explanation so far is a "power event" in one zone of its London cloud region. Its advice to affected customers was to move their workloads to another zone or region "if feasible."
Must be nice. Revolut customers couldn't even move £10.
In July the UK government put Google Cloud, AWS, Microsoft and Oracle under direct Bank of England oversight, because one outage at any of them could shake the entire financial system.
12 days ago the Bank of England ran a drill with the UK's 38 biggest banks and market operators. The scenario they practised was a global cloud outage.
And exactly one year ago today, AWS went down and took Lloyds and Halifax with it.
Same date. Different cloud. Lloyds was having problems again this morning.
Revolut spent five years earning a UK banking licence.
Bitcoin has been running for almost 18 years. It doesn't have a London region.
A special-purpose database optimized for a specific workload can indeed do better than a general multi-purpose one. The hard part is to operate this database with a similar reliability posture.
Should review the cost of operating this and the uptime figures in a year's time.
Full incident write up: buildkitestatus.com/incident…
Highlights from the Aug 25 Buildkite incident:
- Deployment caused an EKS pod to scale up.
- Infra couldn't scale up in time. More scale up requests piled on. New pods put extra pressure on internal DNS.
- Discovery slowed down by 780X and went down with a huge blast radius.
Today was my last day at Amazon after 6+ years. I worked on some of the world's largest scale distributed systems and learnt things I wouldn't be able to imagine before joining Amazon. I'll miss the daily thrill. Next week I'm taking on a new challenge and looking forward to it!



