Wednesday, November 04, 2009

Postmortem on recent issues affecting users in Europe

What happened
Beginning at approximately 2:30am (PDT) Saturday, October 31, a DNS configuration issue in a new datacenter in Europe caused requests to blogger.com and *.blogspot.com to be sent to an invalid IP address. Affected users throughout Europe were unable to connect to either Blogger or to blogs hosted by Blogger at blogspot.com for approximately 10 hours.

What we're doing about it
1. Blogger's systems engineers will proactively monitor and verify DNS settings during major system changes.
2. We have improved our automated monitoring so that on-call engineers will be paged earlier for DNS-related failures.
3. Checks in our DNS management system have been made to prevent the change that caused this to occur.

Eddie Kessler, Blogger Engineering Manager