Thursday, December 28, 2006

Some users are seeing forbidden (403) messages instead of their blogspot blogs. We are aware of this problem and are working on it.

Update, 4:40 am: This issue has been resolved.

Friday, December 15, 2006

Users on the new Blogger are seeing intermittent 500 Server Error pages. We are working to repair this.

Update, 12:01PM: We believe that this problem is taken care of, but we are continuing to monitor closely.

Thursday, December 14, 2006

There will be a scheduled maintenance of Blogger Mobile for unclaimed mobile blogs on the old version Blogger at noon on Tuesday, 12/19/2006, expected to last for the entire day.

Update (12/19/2006 12:05PM): This is now starting.
Update (12/20/2006 16:14PM): This has now finished.

Tuesday, December 12, 2006

Old Blogger is currently down. The new version of Blogger in beta and Blog*Spot are unaffected. We are investigating.

Update, 5PM: This has been fixed.

Friday, December 08, 2006

We are diagnosing and fixing problems with the new version of Blogger in beta right now. Users are seeing 502 server errors when posting to blogs on the beta, and the site’s overall performance is poor.

Update, 3PM: We have fixed the current problem, and are continuing to monitor things closely to prevent relapse. We apologize for the lapse in service.

Saturday, December 02, 2006

Some users on the beta may experience icons disappearing on the post editor and/or trouble uploading images. Cleaning the browser's cache or on Windows force reloading the page by pressing Ctrl-F5 fixes this problem.
There is no planned outage today at 3PM. Some of you may be seeing a warning for it, but that’s because our outage warning system is falling back to a hard-coded default. We’ll get this fixed presently.

Thursday, November 30, 2006

We are currently investigating the intermittent 502 error pages on the new version of Blogger in beta and its Blog*Spot blogs. If you experience one of these errors, waiting a minute or so and refreshing may help.

Update, 11AM Friday: This was cleared up last evening.

Wednesday, November 29, 2006

Until tonight’s mini-outage of the Blogger beta at 8PM PST, the following problems will be seen on the beta:
  • Mail-to-Blogger and Blogger Mobile postings will not publish. They will publish eventually, however, after 8PM.
  • FTP blogs will publish, but the spinny status will not reflect this in the interface. It will instead spin forever.
We apologize for these issues. Everything will be as normal after 8PM.

Update, 8:45PM: The outage completed a little while ago and was mostly successful, with downtime of 5 minutes for the beta and its Blog*Spot. Some blogs were returning 502 error pages for about 15 minutes after the rest of the service returned. All FTP, email, and mobile publishing is now back to normal.

Tuesday, November 28, 2006

There will be a small outage for the new version of Blogger and its Blog*Spot blogs tonight at 7PM, lasting 5 minutes.

Update, 9:45pm: the outage had to be postponed to tomorrow (Wednesday) 10:00am Pacific Time.

Update, 12:42PM Wednesday: This morning’s outage went fine (6 mins for beta, 8 mins for Blog*Spot), though special circumstances knocked this very Status blog (and only Status) for longer than that. There will be a followup outage tonight at 8PM, PST, of roughly 5 minutes.

Wednesday, November 15, 2006

We’ll be doing planned maintenance on the new version of Blogger in beta tomorrow (Thursday). This will require two small outages, one at 12:30PM PST and the other after 3PM PST. Neither outage should last more than 10 minutes. beta.blogger.com and Blog*Spot blogs on the new version of Blogger will be unavailable during the outages. We apologize for these interruptions.

Update, 12:45 Thursday: We've completed the first outage. We've adjusted the time for the second outage to some time after 3PM but before 4PM. Also, the beta and beta blogs will run a little pokey until the second outage.

Update, 1PM Thursday: Non-beta blogs are currently showing 502 server errors. We are investigating this problem and hope to have a fix shortly.

Update, 1:10PM Thursday: Non-beta Blog*Spot is working correctly again. We apologize for this unanticipated side-effect of the planned maintenance.

Update, 3:25PM Thursday: We just completed the second outage. The new version of Blogger is returning to normal. Thanks for your patience surrounding these two brief interruptions, and apologies for the brief unplanned outage of current Blogger's Blog*Spot.

Monday, November 13, 2006

Blog*Spot blogs on the current version of Blogger are showing 500 errors. Our engineers are investigating. Blogger.com and blogs on the new version of Blogger are unaffected.

Update 7:37 PM(PST): This has been fixed.

Sunday, November 12, 2006

Some users on the new version of Blogger in beta are seeing error messages when they publish posts. The posts are still published successfully, however. We have a fix coming, but in the meantime turning off the “Ping Weblogs.com” Publishing setting should prevent the error from occurring.

Update: This problem has been fixed.

Friday, November 03, 2006

Blogger will go down for maintenance on Sunday 4pm PDT. This maintenance will last 45 minutes.

Update: This maintenance is now complete. Thank you for your patience.

Thursday, November 02, 2006

Photo uploading is having problems currently on www.blogger.com. We are working on fixing the problem.
Update 2:45 PM (PST): The issue has been fixed.

Wednesday, November 01, 2006

There are currently problems with photo uploading, we are working on fixing them.

Update 11:40 AM(PST): This issue has been resolved.

Monday, October 30, 2006

We have been working through a series of publishing issues since Saturday morning. The problems have been transient: for some users they were more frequent than for others. We are still working to resolve this completely and will post updates. Blogger beta has not been affected.

Update: this has been fixed.

Sunday, October 29, 2006

There are problems with publishing to blogs. We are working on fixing this. Blogs on the beta are not affected.

Update 9:14 AM (PST):
Publishing works now, expect some slowness though as we wrap up fixing this issue.

Update 16:52 PST: Blogger is inaccessible due to a problematic piece of networking hardware - it should be fixed soon.

Update 17:00 PST: The networking issue has been fixed.

Thursday, October 26, 2006

Sincere apologies for the short notice but we will take down both blogger.com and blogspot.com at 2 PM PST today to replace the piece of network equipment that was causing the outages in the past couple of weeks. The outage will last 60-90 minutes, we will try to get done as fast as we can. Blogger beta will be not affected nor blogs that are hosted on the beta except for a few static images that we pull from www.blogger.com.

Update 3:30 PM (PST): blogger and blogspot are back up (you may need to restart your browser to pick up the DNS change).

Wednesday, October 25, 2006

We experienced an outage this evening due to a problem in our networking hardware, sigh!. This caused our users to lose access to both Blogger and BlogSpot for some time. This has been fixed and things should be back to normal.
We are experiencing some troubles with one filer. This affects publishing to some of the blogs and they will not be available for viewing on blogspot.

Update: The issue has been fixed. Thanks for your patience.

Tuesday, October 24, 2006

Tomorrow we will have an outage of 60 minutes from 2p to 3p (Pacific Time) as we perform some hardware repairs.

Update: This maintenance was completed successfully.

Saturday, October 21, 2006

Wow. Ok. So it hasn’t been Blogger’s weekend. Blogger was down for a little over three hours this evening, due to the near-simultaneous failure of a critical component and its backup. This outage also impacted the loading of many Blog*Spot blogs, which rely on Blogger for a CSS file.

We apologize profusely for this outage. Blogger should be working as normal now. The new version of Blogger in beta was not affected.

Friday, October 20, 2006

We’ve been having a bit of slowness this morning. Technicians are on the scene. Update, 11:15: Most of the problems are resolved. Thanks for sticking with us through the maintenance. Blogger will be running a bit slower than normal for a short time (possibly for the rest of today) as we have a handful of servers out of commission. If you see any error pages, please just wait a sec and try again.

Wednesday, October 18, 2006

Apologies to everyone affected by tonight's Blogger slowness. And super apologies that said slowness prevented us from updating this page in a timely manner.

Monday, October 16, 2006

Some users are experiencing problems with uploaded photos (truncation or not showing at all). We are investigating this problem. If you experience any of these issues please report the blog and image urls to Blogger Support.
Update 9 PM (PST): This problem has been resolved.

Thursday, October 12, 2006

Blogger and Blogspot were down between 5:30 and 7:30 PST this morning due to a network malfunction. We apologize for the inconvenience and the fact that this blog was not updated earlier, but since this blog is also served by Blogger, updating it was of little use.

Sunday, October 01, 2006

Blogger beta is currently unreachable. We are working to fix this issue. We regret the inconvenience caused to our beta users. This does not affect our users on Blogger (http://www.blogger.com).

Update: Blogger beta should be back up for everyone within an hour.

Thursday, September 28, 2006

Blogger's performance has been sluggish this morning. The culprit seems to be a bunch of spammers creating spam blogs at a high-rate. We are locking the spam creators out from Blogger and monitoring the system to see that there are no other issues. Thanks for your patience.

Update 2:55 PM (PST): This issue has been solved. Please report publishing problems to us if you still see them.

Update 10:00 PM (PST): Some users are still experiencing publishing trouble, we are working on figuring this one out. Again, apologies for the the continuing problems.

Update 10:44 PM (PST): We found the cause of the publishing problem, it's those spammers again. Normal publishing will be restored shortly.

Update 11:00 PM (PST): Publishing should now work normally.

Wednesday, September 20, 2006

Blogger will be down for maintenance tomorrow (Sep 21) for 45 minutes starting at 4pm (Pacific Time).

Update: The maintenance was completed successfully.

Monday, September 18, 2006

Blogger will be down for maintenance tomorrow for 15 minutes starting at 4pm (Pacific Time).

Update: The maintenance was completed successfully.

Friday, September 15, 2006

A small percentage of blogs will be down (both viewing and publishing) between 4:10pm and 4:20pm (PDT) as a part of clearing out today morning's blogspot issue.

Update: The downtime will now be sometime between 4:30 pm and 5:30 pm(PDT) for a short while. It should affect a very small percentage of blogs.
Update: The maintenance has been completed successfully.
After this morning's blogspot outage there are still a number of blogs that return an error page. We are working on restoring these blogs. Note that the posts of these blogs are accessible in blogger.com but the the published pages are temporarily down (so we did not lose your posts). We sincerely apologize for the downtown and will send updates as we have more information available.

Update (10:17 PDT): the blogs that had trouble are viewable now. If you still have trouble viewing your blog please notify Blogger support. Again, we apologize for the inconvenience.
Blogspot was down for about 30 minutes. It has been restored now, but some blog pages are still returning an error page. We are working on resolving the problem.

Update, 4:25AM: This issue has been fixed.

Friday, September 08, 2006

We are having some trouble on blogspot: some blogs return an error page. We are working on fixing this.

Update, 3:20PM: This issue has been fixed.

Wednesday, August 30, 2006

Posting to the Blogger Atom API (e.g. via a third-party website or desktop client) is not working properly. We are investigating.

Update, 11:24AM: This has been fixed.

Tuesday, August 29, 2006

One of our database servers has been malfunctioning this morning. We are working on diagnosing the problem.

Update: this issue has been resolved now.

Sunday, August 27, 2006

Some more of those "no space left on device" errors are popping up this morning. We're working on getting them all cleared up right now.

Friday, August 25, 2006

Blogger users are experiencing slowness and occasional timeouts. We are working on diagnosing and fixing the problem.

Update: this problem has been fixed.

Tuesday, August 22, 2006

Some users are seeing a “001 java.io.IOException: No space left on device” error message when publishing to Blog*Spot. We’re on it.

Update, 1PM: This has been fixed.

Tuesday, August 15, 2006

Some password recovery emails are being sent out with incorrect links, which will cause an error when you try to click on them. We're working on getting this fixed soon, but in the meantime if this affects you, you may be able to work around it yourself. Check your link to make sure there is a "/" after the "blogger.com" part, and if it's missing, add it in. E.g.: http://www.blogger.comlotsofcrazytexthere would become http://www.blogger.com/lotsofcrazytexthere.

If that doesn't help, please be patient, and check back on this page for updates. Once we've fixed the problem, you can request a new password recovery email which will work.

Update (2pm PDT): This should now be fixed. If you are still having trouble getting your login information, please request a new password recovery email here.

Tuesday, August 08, 2006

Blogger will be down for maintenance tomorrow for 45 minutes starting at 4pm (Pacific Time).

Update: This has been reschedule to tomorrow (8/10) at 4pm.

Friday, August 04, 2006

We are having problems with one of the Blogger databases. Posts and commments to blogs that are stored on that database are failing. We are working on getting the database back up. Thanks for your patience.

Update (8:45a): this has been fixed.

Tuesday, August 01, 2006

Due to a configuration change made earlier this evening, around 10% of requests made to Blog*Spot will not be successfully returned. Users will see slowness when trying to access a blog on Blog*Spot and a server error will be returned.

If you see this problem, refreshing the page you are trying to access will typically resolve the problem. And we are working on a fix as well.

Update 11:24p: This has now been resolved.
Update 11:56p: To be more precise, it has been resolved but the change has to propagate which may take a little more time.

Thursday, July 27, 2006

Some folks are finding that some of their emailed posts get dropped and don't reach their blogs. We're looking into this.

Update, 8/1: We found the problem in one of our mail servers, so new emailed posts should go through normally now. You might also start seeing old messages show up that previously hadn't gone through.

Tuesday, July 25, 2006

Some users are noticing problems with photo uploading timing out or taking a very long time to respond. We're looking into this and will post an update when there is more information.

Monday, July 24, 2006

We've been having some database issues this morning, causing Blogger.com to be inaccessible for some users. We're investigating the cause of the trouble, so thanks for bearing with us as we work on getting everything running normally again.

Update: For the majority of users things should be working properly again now, though there may still be a few smaller, lingering issues. We plan on having these resolved by the end of the day.

Tuesday, July 18, 2006

We need to perform hardware maintenance on the photos system for Blogger. This means that photo uploading and serving will be down for approximately 1 hour starting at 5pm (Pacific Time). All photos previously uploaded will not be visible during this maintenance.

Thanks for your understanding.

Update: This is now complete. Thanks for your patience.

Sunday, July 16, 2006

Earlier today some users were experiencing trouble publishing their FTP blog. If you were seeing "0% complete" for long periods of time, please try publishing again.

Wednesday, July 12, 2006

Photo uploading and serving will be briefly unavailable for 5 minutes around 2p today. During this time photos will be unviewable as we perform hardware maintenance.

Monday, July 10, 2006

Tomorrow we will have an outage of 90 minutes from 4p to 4:30p 5:30p (Pacific Time) as we perform some hardware upgrades. At the beginning of this outage (the first five minutes), blogger.com will appear completely inaccessible. Following this, a status message will appear for the remainder of the 90 minute outage during which time users will be unable to update their blogs.

Update: The maintenance is now complete and took only 35 minutes. Blogger should be back to normal.

Friday, June 16, 2006

We're currently experiencing some problems with the help site and are working to restore access.

Update: The help site is working normally again. (11:45 AM)

Wednesday, June 14, 2006

We continue to upgrade hardware and make changes in response to last week's problems. Additionally, we are still investigating the photo uploading problem seen by some users. While this problem has been reported by a number of people, it is not consistently reproduceable - we're working with the folks who are running into this to gather as much information as possible about how it occurs.

Friday, June 09, 2006

Late last night we pushed new code and once again replaced harware in order to combat the problems of the past few days. As of this morning it appears to have improved the responsiveness of the site, altho' there are still fixes we want to make to get Blogger faster for everyone.

Also, we are working on fixing the problems some users are seeing with photo upload failures.

Thursday, June 08, 2006

For many users, Blogger will have been extremely slow or down for most of the morning. We continue to work on fixes for this problem and hope to have it resolved as quickly as possible.

Update (4:45p): We are planning another infrastructure overhaul to address the significant problems we've been having in the past several days.

Update (6:59p): We've made another change that has improved performance, but we are also planning to make additional changes this evening.

Update (10:45a, June 9): The issues with the database servers have been resolved last night.

Wednesday, June 07, 2006

This morning, a hardware problem we had been struggling with over the past several days suddenly worsened. As a result, and to stave off future downtime, we have had to take Blogger down for an extended period of time as we address the problem. We will continue to update this blog and the homepage with more information.

Update: We've brought up new hardware which has allowed us to restore the site. We try hard to avoid downtimes of this length and apologize for the inconvenience.

Monday, June 05, 2006

Photo uploading will be disabled for 30 minutes this evening starting at 10pm (Pacific Time).

Update: This is taking longer than expected. Hopefully we will be done by 11:45pm (Pacific Time). We will update as we learn more.

Update: We are now finished. Thank you for your patience.
We are now recovering from an unscheduled downtime, related to the problem from last week.

There is still some residual slowness as a result of emergency maintenance we are performing on the system.

Friday, June 02, 2006

We are having intermittent downtime because of a hardware problem. We are working to resolve this issue.

Update: This has been resolved.

Wednesday, May 31, 2006

Some users will be seeing an error at login because of a change made last night. We are working on a fix for this problem now.

Update, 10AM Thursday: This was fixed at about 2PM yesterday afternoon.

Saturday, May 27, 2006

From approximately 5:45pm to 6:30pm, some people may have had trouble accessing their blogs as a database job was causing queries to be slow. Things have been restored to normal. Sorry for the interruption.

Friday, May 26, 2006

Right now we’re having problems with blog publishing. Affected users will see the publish status page hang at 0% complete. Our engineers are investigating.

Update, 3:55: All seems to be back to normal. If you weren’t able to publish before, go to the “Status” subtab and republish. No posts were lost. Apologies to all of you who were affected, and to your readers as well.

Tuesday, May 09, 2006

A networking issue caused both Blogger and Blog*Spot to be sluggish and and partially unresponsive for a period of about 2 hours this morning. We are now recovered from this outage.

Monday, May 01, 2006

Currently some users are getting “No route to host” errors when publishing to Blog*Spot. We’re looking into this. In the mean time, logging out and clearing cookies should relieve the problem.

Update, 6:45PM: All sorted out. Apologies to those folks who were affected by this during the afternoon.

Monday, April 24, 2006

Blogger.com will be down on April 24 2006 from 4 pm PDT to 4:45 pm PDT due to planned maintenance. We’re sorry about the one-two unplanned/planned outage punch today, but we need to do some database maintenance. You will still be able to view your blogs during the outage. Be assured that when Blogger.com comes back, it will be shinier and happier than ever.

Update: The maintenance is complete. Blogger is back to normal.
All publishing is broken right now. We’re working on fixing it.

Update, 10:15AM: We have Blog*Spot publishing working again. External publishing coming soon.

Update, 10:41AM: External publishing is working again as well. Plus users, we haven't forgotten about you.

Update, 10:50AM: Everything sorted out now and working fine. Expect possible transient slownesses as we shore up some of the quick fixes that we had to make.

Friday, April 21, 2006

One of our databases is down, which prevents users from publishing to certain blogs. We are working on getting the database back.

Update:
the database was repaired early this morning PST.

Monday, April 17, 2006

Blogger will be down on April 19 2006 from 4 pm PST to 4:45 pm PST due to planned maintenance. We regret any inconvenience caused to our users. You will still be able to view your blogs.

(Correction: Whoops! Technically, we meant “PDT,” not “PST.” Daylight saving and all that. Hope we didn’t throw anyone.)

Saturday, April 15, 2006


At 11:30 this morning, we experienced a system failure in one of our databases and some people were unable to create new entries on their blog during that time. As of now that database is back online and those users are again able to create new entries on their blog.



We are investigating what failed and will take measures to make sure it doesn't happen again.

Saturday, April 08, 2006

Blogger was slower than normal this morning due to database issues. We performed some quick maintenance and things should be back to normal now.

Wednesday, April 05, 2006

Blogger will be down today from 5 pm PST for 45 minutes. We regret any inconvenience caused to our users. You will still be able to view your blogs.

Update: The maintenance was successful. Blogger is back to normal.

Tuesday, April 04, 2006

Some users will be unable to view or publish their blogs for half an hour between 4 pm and 4:30 pm PST. We have to do some maintenance one of our filers. Sorry for the inconvenience.

Sunday, April 02, 2006

Some users are experiencing problems creating new posts via our Atom API. We are working to get this fixed. In the meantime, you can create new posts via the web interface.


Update 4:30pm This is now fixed. Sorry for the trouble.

Monday, March 27, 2006

We’re experiencing some intermittent slowness in our photo uploading infrastructure. If you’re patient, your photo may get uploaded before your browser times out. Folks are investigating and fixing.

Update, 9:15: This has been fixed.

Wednesday, March 22, 2006

Some of you may be seeing the “Blogger Problem” error page when you visit Blogger.com. If you see this, clearing your blogger.com cookies and trying again will resolve it (might take a few tries, though). Some of the servers are misbehaving, but we’re getting it all back in line now.

Update, 5:25: All sorted out.
We've finally finished moving all the affected blogs off of the bad machine that has been causing us so much trouble recently. You may see some lingering problems with a few blogs that were incorrectly or incompletely published on their last update, but republishing any such blogs should fix them. If you continue to receive errors viewing any blogs, try clearing your browser's cache and then refreshing the page. Going forward, though, things should be back to working normally. Thanks for sticking with us through this.

Saturday, March 18, 2006

A clarification about the filer we restored yesterday: This machine is indeed up and functioning again, so the affected blogs are no longer entirely inaccessible. However, it is still not in great shape and we are in the process of moving all the data off of it and on to better machines. So over the next few days there may still be lingering and intermittent problems for some blogs. This includes the "forbidden" errors we're all getting tired of, as well as occasional publishing errors, or incompletely published pages. If you get an error viewing a blog, refreshing the page once or twice should clear it. For publishing problems, simply wait a few minutes and republish, and that should take care of it. Thanks for your patience while we work on clearing all this up.

Thursday, March 16, 2006

The filer that we have been having trouble with in the last few days failed again. Those blogs that are stored on the bad filer are temporarily not available for publishing and viewing. We are working on replacing the filer and restoring access to the blogs affected.

Update (10:40 am, March 17): The filer has been restored. All affected blogs are available for publishing and reading.
Currently photo uploading is not working for FTP users. We are aware of the problem and should have a fix within a few hours. We apologize for the inconvenience.


Update (4:10pm) : This has been fixed. Thank you for your patience.

Unfortunately, we have to do some maintenance on one of our filers today from 9 am to 9:40 am PST. Some blogs will be inaccessible while this is going on. Everyone is able to publish, however, regardless of whether or not the blog is down for maintenance.

We apologize to those blog owners who are affected by this.

Update: This maintenance is now complete.

Wednesday, March 15, 2006

We’re doing some maintenance on one of our Blog*Spot servers. Some blogs will be inaccessible while this is going on. Everyone is able to publish, however, regardless of whether or not the blog is down for maintenance.

We apologize to those blog owners who are affected by this. This partial outage is necessary to fix some of the transient Blog*Spot problems that have popped up recently.

Update: The maintenance is complete.

Wednesday, March 08, 2006

Some users are currently getting “403 Forbidden” errors when viewing their blogs, and “Permission Denied” errors (or variations thereof) when publishing. You may recognize this problem from Monday.

The offending server is being replaced and then shot. We’ll let you know when things are back up. Shouldn’t be too long.

Until then, affected blogs will be unavailable and unpublishtoable.

Update, 5 minutes later: All fixed!

Monday, March 06, 2006

Some users are currently getting "403 Forbidden" errors when viewing their blogs, and "Permission Denied" errors (or variations thereof) when publishing. We're restarting some machines right now and things should be working again very soon.

Update: This has now been fixed. If you still encounter problems, please clear your browser's cache and cookies, then give it another shot.

Saturday, February 18, 2006

We’re experiencing some problems with one of our databases. Some posts may not be saved, and other, saved posts, may not be visible in the app. Working on fixing this now…

Update, 12:44PM: This has been sorted out. We’re continuing to investigate the root cause and why our monitoring didn’t catch it sooner, but Blogger is back to functioning normally. Posts made from now on will be saved.

I’m very sorry to say that, if your blog was on this database, posts and template changes made in the last 18 hours or so were not saved. They may appear on your blog now, but will disappear if you republish. If you made a post between Friday afternoon and now, we suggest that you look at your list of posts (“Posting” tab, “Edit posts” sub-tab) and compare it with what is published on your blog. If posts are missing, copy them from your blog pages before you republish.

Major apologies to those of you who were affected. We work hard to maintain Blogger as a trustworthy place to keep your writing, and we really hate to let you down.

Thursday, February 16, 2006

A small number of users are seeing a 553 Permission denied error when publishing to Blog*Spot. If this happens to you, logout of Blogger, clear your blogger.com cookies, and log back in. We are working to fix this problem.

Update, 2/21: This is now fixed.

Thursday, February 09, 2006

Sorry for the short notice. Blogger.com will be down tonight, Thursday, February 9th, from 9:00–10:00PM PST. Blog*Spot will be unaffected.

We’re upgrading a database, which gives us more room to grow and will fix some transient comment problems that some of you have had recently.

Monday, February 06, 2006

The domain association problem mentioned yesterday has now been resolved. If you are still experiencing a problem with your mapped domain, please let Support know.

A reminder that we will also be down for planned maintenance starting at 7p (PST) this evening. We will be using this time to replace the older hardware responsible for many of the problems we experienced last week.

Sunday, February 05, 2006

Just a quick reminder that we will be going ahead with a planned network maintenance on Monday the 6th from 7–8PM PST. Blogger and Blog*Spot blogs will be unavailable during that time.

This maintenance won’t fix everything, but it will make things better. I promise.

Saturday, February 04, 2006

As many people will have noticed, we've had significant hardware trouble over the past couple days. We have tried to keep all of you informed about the status of the service and our efforts to correct the problems. Unfortunately, today's problems were widespread enough such that we were unable to provide ongoing updates. One of the things we'll be doing in the short term is to come up with ways to prevent this.

Eric details the chronology below. There is one residual issue related to the outage: some legacy users have domain associations between Blog*Spot and external domains. These associations are not functional at this time.

In addition to working on this problem, we'll also be making the necessary changes to guard against the outages we experienced over the past 48 hours.

We will provide updates on our progress as changes are made. I sincerely apologize for the downtime - in the past six months I feel we've made significant strides to limit this type of failure. We obviously have more work to do and carrying it out will be our highest priority.
Blogspot is again experiencing problems - we are investigating.

Update, 8PM: We have restored all of Blog*Spot, save one of our filers. This means that some blogs will still be unpublishable and inaccessible. Our engineers are continuing to work on this problem.

Update, 11PM: Blog*Spot servers are restarting now and connecting with the filer. All blogs should be publishable and accessible within the next 20–30 minutes.

[So how has everyone else’s weekend been? —Pete]

Friday, February 03, 2006

We are currently experiencing some problems with Blog*Spot. A percentage of blogs are completely inaccessible. For others, you may need to reload a few times before you connect to a web server that will serve them. We anticipate things will be calmed down within the hour.

Update, 5:20PM: Blogger and Blog*Spot should be fully operational with the exception of one of our file servers. This means a percentage of blogs will be unpublishable and inaccessible. We are currently waiting for a lengthy disk integrity check to complete on the file server. There may also be transient delays serving blog pages while our load balancing evens out.

Update, 8:00PM: The Blog*Spot web servers are coming back online with connections to the file server that was out for most of the evening. (Its disk is totally okay, btw.) So, within the next 10–15 minutes everything should be completely back to normal.

Again, super major apologies for this.
Blogger.com is back working again, again. Unscheduled outage hat trick tonight. As I get more information from our worldwide operations team I’ll let you know a bit more of why Blogger.com went down, and what we’ve done to keep this particular problem at bay.
Blogger.com is now back from a 45 minute outage. Two outages in one night. I feel like Dante from Clerks: “I’m not even supposed to be carrying the pager today!”

We’re really sorry about this spate of random downtime. As Jason mentioned, I mentioned that Monday’s fixup will lessen these types of outages in the future. The Blogger and Google engineers and ops folks are not just sitting around waiting for the next failure; we’re actively improving our infrastructure to lessen both the planned and unplanned Blogger outages.

Thursday, February 02, 2006

We're recovering now from a unscheduled outage of about 20 minutes. As Pete mentions below, we are almost finished migrating from some unreliable hardware. This should help to prevent downtimes like this.

Wednesday, February 01, 2006

Okay, big lead time on this one. Lots of time to prepare yourself.

Blogger and Blog*Spot will be offline for approximately one hour from 7–8PM this coming Monday, February 6th. The little outage indicator will kick in Sunday night to remind you.

This time, we’re completing the long–prepared for cutover from an older, flakier piece of network equiptment to a shiny bright new one. (This is how they explain things to software guys like me.) That means fewer planned and “unplanned” outages in the future. Seriously. I wouldn’t be too surprised if the ops guys get all Office Space on the thing they’re taking out, given the trouble it’s caused.

Now get thee back to blogging.

Thursday, January 26, 2006

We're now recovering from a 15 minute outage. Apologies for the unscheduled downtime.

Tuesday, January 24, 2006

We'll be taking Blogger down on Wednesday the 25th at 4pm PST to fix a bit of a switch that's gone wonky on us. The outage should last about 15 minutes. Blogger.com and Blog*Spot blogs will be inaccessible during this time.

This repair will fix the problem that caused the brief outage last Friday night. We're also using this down time as an opportunity to tune our databases for more efficient spam catching and deletion.

Update:
Theres a notice saying "scheduled outage at 4:00 PST" on blogger. I spent the whole time of writing this trying to figure out what they might do to cause a scheduled outrage before I noticed the lack of an r.
Found on McBlog

(P.S.: The switch and the rest of Blogger is cheerful and productive.)

Monday, January 16, 2006

Over the weekend, some users were seeing generic "unexpected error" messages when attempting to log in to their accounts. That problem has been patched as of about 10 AM this morning (PST), so if you were affected by it your account should be working normally again now. If you still see the error, just clear your browser's cookies, then log in again.
At 6am, some Blogger users suffered a 10-minute outage while a database took a nap. We apologize for the inconvenience while we continue to improve our service.

Update: After some early morning investigation, we further pinned down and isolated the issue. We will be taking steps this week to ensure this particular problem does not occur again.

Thursday, January 12, 2006

Blogger will be down for ~10 minutes this afternoon at 3:30pm PST, for some network maintenance.

Update: This outage has been cancelled.

Tuesday, January 03, 2006

Some users are seeing "No space left on device" when attempting to publish. We're looking into this error and hope to have it resolved shortly.

Status: This has been resolved.