staff:

This morning, our primary datacenter effectively lost its connection to the Internet after our upstream provider (ISP) suffered a major outage caused by broken fiber lines.

It took nearly five hours before service was fully restored.

Our failover procedures, designed to immediately rebalance…

Me: So it was partially Tumblr’s fault for not preparing for this scenario, but the whole thing was caused by an Internet Service Provider’s fiber being cut.

I actually had this happen before at a company I worked for (construction crew cut the fiber) so I can sympathize with their situation.