Some users are seeing forbidden (403) messages instead of their blogspot blogs. We are aware of this problem and are working on it.
Update, 4:40 am: This issue has been resolved.
Thursday, December 28, 2006
Friday, December 15, 2006
Thursday, December 14, 2006
Tuesday, December 12, 2006
Friday, December 08, 2006
We are diagnosing and fixing problems with the new version of Blogger in beta right now. Users are seeing 502 server errors when posting to blogs on the beta, and the site’s overall performance is poor.
Update, 3PM: We have fixed the current problem, and are continuing to monitor things closely to prevent relapse. We apologize for the lapse in service.
Update, 3PM: We have fixed the current problem, and are continuing to monitor things closely to prevent relapse. We apologize for the lapse in service.
Saturday, December 02, 2006
Some users on the beta may experience icons disappearing on the post editor and/or trouble uploading images. Cleaning the browser's cache or on Windows force reloading the page by pressing Ctrl-F5 fixes this problem.
Thursday, November 30, 2006
Wednesday, November 29, 2006
Until tonight’s mini-outage of the Blogger beta at 8PM PST, the following problems will be seen on the beta:
Update, 8:45PM: The outage completed a little while ago and was mostly successful, with downtime of 5 minutes for the beta and its Blog*Spot. Some blogs were returning 502 error pages for about 15 minutes after the rest of the service returned. All FTP, email, and mobile publishing is now back to normal.
- Mail-to-Blogger and Blogger Mobile postings will not publish. They will publish eventually, however, after 8PM.
- FTP blogs will publish, but the spinny status will not reflect this in the interface. It will instead spin forever.
Update, 8:45PM: The outage completed a little while ago and was mostly successful, with downtime of 5 minutes for the beta and its Blog*Spot. Some blogs were returning 502 error pages for about 15 minutes after the rest of the service returned. All FTP, email, and mobile publishing is now back to normal.
Tuesday, November 28, 2006
There will be a small outage for the new version of Blogger and its Blog*Spot blogs tonight at 7PM, lasting 5 minutes.
Update, 9:45pm: the outage had to be postponed to tomorrow (Wednesday) 10:00am Pacific Time.
Update, 12:42PM Wednesday: This morning’s outage went fine (6 mins for beta, 8 mins for Blog*Spot), though special circumstances knocked this very Status blog (and only Status) for longer than that. There will be a followup outage tonight at 8PM, PST, of roughly 5 minutes.
Update, 9:45pm: the outage had to be postponed to tomorrow (Wednesday) 10:00am Pacific Time.
Update, 12:42PM Wednesday: This morning’s outage went fine (6 mins for beta, 8 mins for Blog*Spot), though special circumstances knocked this very Status blog (and only Status) for longer than that. There will be a followup outage tonight at 8PM, PST, of roughly 5 minutes.
Wednesday, November 15, 2006
We’ll be doing planned maintenance on the new version of Blogger in beta tomorrow (Thursday). This will require two small outages, one at 12:30PM PST and the other after 3PM PST. Neither outage should last more than 10 minutes. beta.blogger.com and Blog*Spot blogs on the new version of Blogger will be unavailable during the outages. We apologize for these interruptions.
Update, 12:45 Thursday: We've completed the first outage. We've adjusted the time for the second outage to some time after 3PM but before 4PM. Also, the beta and beta blogs will run a little pokey until the second outage.
Update, 1PM Thursday: Non-beta blogs are currently showing 502 server errors. We are investigating this problem and hope to have a fix shortly.
Update, 1:10PM Thursday: Non-beta Blog*Spot is working correctly again. We apologize for this unanticipated side-effect of the planned maintenance.
Update, 3:25PM Thursday: We just completed the second outage. The new version of Blogger is returning to normal. Thanks for your patience surrounding these two brief interruptions, and apologies for the brief unplanned outage of current Blogger's Blog*Spot.
Update, 12:45 Thursday: We've completed the first outage. We've adjusted the time for the second outage to some time after 3PM but before 4PM. Also, the beta and beta blogs will run a little pokey until the second outage.
Update, 1PM Thursday: Non-beta blogs are currently showing 502 server errors. We are investigating this problem and hope to have a fix shortly.
Update, 1:10PM Thursday: Non-beta Blog*Spot is working correctly again. We apologize for this unanticipated side-effect of the planned maintenance.
Update, 3:25PM Thursday: We just completed the second outage. The new version of Blogger is returning to normal. Thanks for your patience surrounding these two brief interruptions, and apologies for the brief unplanned outage of current Blogger's Blog*Spot.
Monday, November 13, 2006
Sunday, November 12, 2006
Some users on the new version of Blogger in beta are seeing error messages when they publish posts. The posts are still published successfully, however. We have a fix coming, but in the meantime turning off the “Ping Weblogs.com” Publishing setting should prevent the error from occurring.
Update: This problem has been fixed.
Update: This problem has been fixed.
Friday, November 03, 2006
Thursday, November 02, 2006
Wednesday, November 01, 2006
Monday, October 30, 2006
We have been working through a series of publishing issues since Saturday morning. The problems have been transient: for some users they were more frequent than for others. We are still working to resolve this completely and will post updates. Blogger beta has not been affected.
Update: this has been fixed.
Update: this has been fixed.
Sunday, October 29, 2006
There are problems with publishing to blogs. We are working on fixing this. Blogs on the beta are not affected.
Update 9:14 AM (PST): Publishing works now, expect some slowness though as we wrap up fixing this issue.
Update 16:52 PST: Blogger is inaccessible due to a problematic piece of networking hardware - it should be fixed soon.
Update 17:00 PST: The networking issue has been fixed.
Update 9:14 AM (PST): Publishing works now, expect some slowness though as we wrap up fixing this issue.
Update 16:52 PST: Blogger is inaccessible due to a problematic piece of networking hardware - it should be fixed soon.
Update 17:00 PST: The networking issue has been fixed.
Thursday, October 26, 2006
Sincere apologies for the short notice but we will take down both blogger.com and blogspot.com at 2 PM PST today to replace the piece of network equipment that was causing the outages in the past couple of weeks. The outage will last 60-90 minutes, we will try to get done as fast as we can. Blogger beta will be not affected nor blogs that are hosted on the beta except for a few static images that we pull from www.blogger.com.
Update 3:30 PM (PST): blogger and blogspot are back up (you may need to restart your browser to pick up the DNS change).
Update 3:30 PM (PST): blogger and blogspot are back up (you may need to restart your browser to pick up the DNS change).
Wednesday, October 25, 2006
Tuesday, October 24, 2006
Saturday, October 21, 2006
Wow. Ok. So it hasn’t been Blogger’s weekend. Blogger was down for a little over three hours this evening, due to the near-simultaneous failure of a critical component and its backup. This outage also impacted the loading of many Blog*Spot blogs, which rely on Blogger for a CSS file.
We apologize profusely for this outage. Blogger should be working as normal now. The new version of Blogger in beta was not affected.
We apologize profusely for this outage. Blogger should be working as normal now. The new version of Blogger in beta was not affected.
Friday, October 20, 2006
We’ve been having a bit of slowness this morning. Technicians are on the scene. Update, 11:15: Most of the problems are resolved. Thanks for sticking with us through the maintenance. Blogger will be running a bit slower than normal for a short time (possibly for the rest of today) as we have a handful of servers out of commission. If you see any error pages, please just wait a sec and try again.
Wednesday, October 18, 2006
Monday, October 16, 2006
Thursday, October 12, 2006
Sunday, October 01, 2006
Blogger beta is currently unreachable. We are working to fix this issue. We regret the inconvenience caused to our beta users. This does not affect our users on Blogger (http://www.blogger.com).
Update: Blogger beta should be back up for everyone within an hour.
Update: Blogger beta should be back up for everyone within an hour.
Thursday, September 28, 2006
Blogger's performance has been sluggish this morning. The culprit seems to be a bunch of spammers creating spam blogs at a high-rate. We are locking the spam creators out from Blogger and monitoring the system to see that there are no other issues. Thanks for your patience.
Update 2:55 PM (PST): This issue has been solved. Please report publishing problems to us if you still see them.
Update 10:00 PM (PST): Some users are still experiencing publishing trouble, we are working on figuring this one out. Again, apologies for the the continuing problems.
Update 10:44 PM (PST): We found the cause of the publishing problem, it's those spammers again. Normal publishing will be restored shortly.
Update 11:00 PM (PST): Publishing should now work normally.
Update 2:55 PM (PST): This issue has been solved. Please report publishing problems to us if you still see them.
Update 10:00 PM (PST): Some users are still experiencing publishing trouble, we are working on figuring this one out. Again, apologies for the the continuing problems.
Update 10:44 PM (PST): We found the cause of the publishing problem, it's those spammers again. Normal publishing will be restored shortly.
Update 11:00 PM (PST): Publishing should now work normally.
Wednesday, September 20, 2006
Monday, September 18, 2006
Friday, September 15, 2006
A small percentage of blogs will be down (both viewing and publishing) between 4:10pm and 4:20pm (PDT) as a part of clearing out today morning's blogspot issue.
Update: The downtime will now be sometime between 4:30 pm and 5:30 pm(PDT) for a short while. It should affect a very small percentage of blogs.
Update: The maintenance has been completed successfully.
Update: The downtime will now be sometime between 4:30 pm and 5:30 pm(PDT) for a short while. It should affect a very small percentage of blogs.
Update: The maintenance has been completed successfully.
After this morning's blogspot outage there are still a number of blogs that return an error page. We are working on restoring these blogs. Note that the posts of these blogs are accessible in blogger.com but the the published pages are temporarily down (so we did not lose your posts). We sincerely apologize for the downtown and will send updates as we have more information available.
Update (10:17 PDT): the blogs that had trouble are viewable now. If you still have trouble viewing your blog please notify Blogger support. Again, we apologize for the inconvenience.
Update (10:17 PDT): the blogs that had trouble are viewable now. If you still have trouble viewing your blog please notify Blogger support. Again, we apologize for the inconvenience.
Friday, September 08, 2006
Wednesday, August 30, 2006
Tuesday, August 29, 2006
Sunday, August 27, 2006
Friday, August 25, 2006
Tuesday, August 22, 2006
Tuesday, August 15, 2006
Some password recovery emails are being sent out with incorrect links, which will cause an error when you try to click on them. We're working on getting this fixed soon, but in the meantime if this affects you, you may be able to work around it yourself. Check your link to make sure there is a "/" after the "blogger.com" part, and if it's missing, add it in. E.g.:
If that doesn't help, please be patient, and check back on this page for updates. Once we've fixed the problem, you can request a new password recovery email which will work.
Update (2pm PDT): This should now be fixed. If you are still having trouble getting your login information, please request a new password recovery email here.
http://www.blogger.comlotsofcrazytexthere
would become http://www.blogger.com/lotsofcrazytexthere
. If that doesn't help, please be patient, and check back on this page for updates. Once we've fixed the problem, you can request a new password recovery email which will work.
Update (2pm PDT): This should now be fixed. If you are still having trouble getting your login information, please request a new password recovery email here.
Tuesday, August 08, 2006
Friday, August 04, 2006
Tuesday, August 01, 2006
Due to a configuration change made earlier this evening, around 10% of requests made to Blog*Spot will not be successfully returned. Users will see slowness when trying to access a blog on Blog*Spot and a server error will be returned.
If you see this problem, refreshing the page you are trying to access will typically resolve the problem. And we are working on a fix as well.
Update 11:24p: This has now been resolved.
Update 11:56p: To be more precise, it has been resolved but the change has to propagate which may take a little more time.
If you see this problem, refreshing the page you are trying to access will typically resolve the problem. And we are working on a fix as well.
Update 11:24p: This has now been resolved.
Update 11:56p: To be more precise, it has been resolved but the change has to propagate which may take a little more time.
Thursday, July 27, 2006
Some folks are finding that some of their emailed posts get dropped and don't reach their blogs. We're looking into this.
Update, 8/1: We found the problem in one of our mail servers, so new emailed posts should go through normally now. You might also start seeing old messages show up that previously hadn't gone through.
Update, 8/1: We found the problem in one of our mail servers, so new emailed posts should go through normally now. You might also start seeing old messages show up that previously hadn't gone through.
Tuesday, July 25, 2006
Monday, July 24, 2006
We've been having some database issues this morning, causing Blogger.com to be inaccessible for some users. We're investigating the cause of the trouble, so thanks for bearing with us as we work on getting everything running normally again.
Update: For the majority of users things should be working properly again now, though there may still be a few smaller, lingering issues. We plan on having these resolved by the end of the day.
Update: For the majority of users things should be working properly again now, though there may still be a few smaller, lingering issues. We plan on having these resolved by the end of the day.
Tuesday, July 18, 2006
We need to perform hardware maintenance on the photos system for Blogger. This means that photo uploading and serving will be down for approximately 1 hour starting at 5pm (Pacific Time). All photos previously uploaded will not be visible during this maintenance.
Thanks for your understanding.
Update: This is now complete. Thanks for your patience.
Thanks for your understanding.
Update: This is now complete. Thanks for your patience.
Sunday, July 16, 2006
Wednesday, July 12, 2006
Monday, July 10, 2006
Tomorrow we will have an outage of 90 minutes from 4p to 4:30p 5:30p (Pacific Time) as we perform some hardware upgrades. At the beginning of this outage (the first five minutes), blogger.com will appear completely inaccessible. Following this, a status message will appear for the remainder of the 90 minute outage during which time users will be unable to update their blogs.
Update: The maintenance is now complete and took only 35 minutes. Blogger should be back to normal.
Update: The maintenance is now complete and took only 35 minutes. Blogger should be back to normal.
Friday, June 16, 2006
Wednesday, June 14, 2006
We continue to upgrade hardware and make changes in response to last week's problems. Additionally, we are still investigating the photo uploading problem seen by some users. While this problem has been reported by a number of people, it is not consistently reproduceable - we're working with the folks who are running into this to gather as much information as possible about how it occurs.
Friday, June 09, 2006
Late last night we pushed new code and once again replaced harware in order to combat the problems of the past few days. As of this morning it appears to have improved the responsiveness of the site, altho' there are still fixes we want to make to get Blogger faster for everyone.
Also, we are working on fixing the problems some users are seeing with photo upload failures.
Also, we are working on fixing the problems some users are seeing with photo upload failures.
Thursday, June 08, 2006
For many users, Blogger will have been extremely slow or down for most of the morning. We continue to work on fixes for this problem and hope to have it resolved as quickly as possible.
Update (4:45p): We are planning another infrastructure overhaul to address the significant problems we've been having in the past several days.
Update (6:59p): We've made another change that has improved performance, but we are also planning to make additional changes this evening.
Update (10:45a, June 9): The issues with the database servers have been resolved last night.
Update (4:45p): We are planning another infrastructure overhaul to address the significant problems we've been having in the past several days.
Update (6:59p): We've made another change that has improved performance, but we are also planning to make additional changes this evening.
Update (10:45a, June 9): The issues with the database servers have been resolved last night.
Wednesday, June 07, 2006
This morning, a hardware problem we had been struggling with over the past several days suddenly worsened. As a result, and to stave off future downtime, we have had to take Blogger down for an extended period of time as we address the problem. We will continue to update this blog and the homepage with more information.
Update: We've brought up new hardware which has allowed us to restore the site. We try hard to avoid downtimes of this length and apologize for the inconvenience.
Update: We've brought up new hardware which has allowed us to restore the site. We try hard to avoid downtimes of this length and apologize for the inconvenience.
Monday, June 05, 2006
Friday, June 02, 2006
Wednesday, May 31, 2006
Saturday, May 27, 2006
Friday, May 26, 2006
Right now we’re having problems with blog publishing. Affected users will see the publish status page hang at 0% complete. Our engineers are investigating.
Update, 3:55: All seems to be back to normal. If you weren’t able to publish before, go to the “Status” subtab and republish. No posts were lost. Apologies to all of you who were affected, and to your readers as well.
Update, 3:55: All seems to be back to normal. If you weren’t able to publish before, go to the “Status” subtab and republish. No posts were lost. Apologies to all of you who were affected, and to your readers as well.
Tuesday, May 09, 2006
Monday, May 01, 2006
Monday, April 24, 2006
Blogger.com will be down on April 24 2006 from 4 pm PDT to 4:45 pm PDT due to planned maintenance. We’re sorry about the one-two unplanned/planned outage punch today, but we need to do some database maintenance. You will still be able to view your blogs during the outage. Be assured that when Blogger.com comes back, it will be shinier and happier than ever.
Update: The maintenance is complete. Blogger is back to normal.
Update: The maintenance is complete. Blogger is back to normal.
All publishing is broken right now. We’re working on fixing it.
Update, 10:15AM: We have Blog*Spot publishing working again. External publishing coming soon.
Update, 10:41AM: External publishing is working again as well. Plus users, we haven't forgotten about you.
Update, 10:50AM: Everything sorted out now and working fine. Expect possible transient slownesses as we shore up some of the quick fixes that we had to make.
Update, 10:15AM: We have Blog*Spot publishing working again. External publishing coming soon.
Update, 10:41AM: External publishing is working again as well. Plus users, we haven't forgotten about you.
Update, 10:50AM: Everything sorted out now and working fine. Expect possible transient slownesses as we shore up some of the quick fixes that we had to make.
Friday, April 21, 2006
Monday, April 17, 2006
Blogger will be down on April 19 2006 from 4 pm PST to 4:45 pm PST due to planned maintenance. We regret any inconvenience caused to our users. You will still be able to view your blogs.
(Correction: Whoops! Technically, we meant “PDT,” not “PST.” Daylight saving and all that. Hope we didn’t throw anyone.)
(Correction: Whoops! Technically, we meant “PDT,” not “PST.” Daylight saving and all that. Hope we didn’t throw anyone.)
Saturday, April 15, 2006
At 11:30 this morning, we experienced a system failure in one of our databases and some people were unable to create new entries on their blog during that time. As of now that database is back online and those users are again able to create new entries on their blog.
We are investigating what failed and will take measures to make sure it doesn't happen again.
Saturday, April 08, 2006
Wednesday, April 05, 2006
Tuesday, April 04, 2006
Sunday, April 02, 2006
Some users are experiencing problems creating new posts via our Atom API. We are working to get this fixed. In the meantime, you can create new posts via the web interface.
Update 4:30pm This is now fixed. Sorry for the trouble.
Monday, March 27, 2006
Wednesday, March 22, 2006
Some of you may be seeing the “Blogger Problem” error page when you visit Blogger.com. If you see this, clearing your blogger.com cookies and trying again will resolve it (might take a few tries, though). Some of the servers are misbehaving, but we’re getting it all back in line now.
Update, 5:25: All sorted out.
Update, 5:25: All sorted out.
We've finally finished moving all the affected blogs off of the bad machine that has been causing us so much trouble recently. You may see some lingering problems with a few blogs that were incorrectly or incompletely published on their last update, but republishing any such blogs should fix them. If you continue to receive errors viewing any blogs, try clearing your browser's cache and then refreshing the page. Going forward, though, things should be back to working normally. Thanks for sticking with us through this.
Saturday, March 18, 2006
A clarification about the filer we restored yesterday: This machine is indeed up and functioning again, so the affected blogs are no longer entirely inaccessible. However, it is still not in great shape and we are in the process of moving all the data off of it and on to better machines. So over the next few days there may still be lingering and intermittent problems for some blogs. This includes the "forbidden" errors we're all getting tired of, as well as occasional publishing errors, or incompletely published pages. If you get an error viewing a blog, refreshing the page once or twice should clear it. For publishing problems, simply wait a few minutes and republish, and that should take care of it. Thanks for your patience while we work on clearing all this up.
Thursday, March 16, 2006
The filer that we have been having trouble with in the last few days failed again. Those blogs that are stored on the bad filer are temporarily not available for publishing and viewing. We are working on replacing the filer and restoring access to the blogs affected.
Update (10:40 am, March 17): The filer has been restored. All affected blogs are available for publishing and reading.
Update (10:40 am, March 17): The filer has been restored. All affected blogs are available for publishing and reading.
Unfortunately, we have to do some maintenance on one of our filers today from 9 am to 9:40 am PST. Some blogs will be inaccessible while this is going on. Everyone is able to publish, however, regardless of whether or not the blog is down for maintenance.
We apologize to those blog owners who are affected by this.
Update: This maintenance is now complete.
We apologize to those blog owners who are affected by this.
Update: This maintenance is now complete.
Wednesday, March 15, 2006
We’re doing some maintenance on one of our Blog*Spot servers. Some blogs will be inaccessible while this is going on. Everyone is able to publish, however, regardless of whether or not the blog is down for maintenance.
We apologize to those blog owners who are affected by this. This partial outage is necessary to fix some of the transient Blog*Spot problems that have popped up recently.
Update: The maintenance is complete.
We apologize to those blog owners who are affected by this. This partial outage is necessary to fix some of the transient Blog*Spot problems that have popped up recently.
Update: The maintenance is complete.
Wednesday, March 08, 2006
Some users are currently getting “403 Forbidden” errors when viewing their blogs, and “Permission Denied” errors (or variations thereof) when publishing. You may recognize this problem from Monday.
The offending server is being replaced and then shot. We’ll let you know when things are back up. Shouldn’t be too long.
Until then, affected blogs will be unavailable and unpublishtoable.
Update, 5 minutes later: All fixed!
The offending server is being replaced and then shot. We’ll let you know when things are back up. Shouldn’t be too long.
Until then, affected blogs will be unavailable and unpublishtoable.
Update, 5 minutes later: All fixed!
Monday, March 06, 2006
Some users are currently getting "403 Forbidden" errors when viewing their blogs, and "Permission Denied" errors (or variations thereof) when publishing. We're restarting some machines right now and things should be working again very soon.
Update: This has now been fixed. If you still encounter problems, please clear your browser's cache and cookies, then give it another shot.
Update: This has now been fixed. If you still encounter problems, please clear your browser's cache and cookies, then give it another shot.
Saturday, February 18, 2006
We’re experiencing some problems with one of our databases. Some posts may not be saved, and other, saved posts, may not be visible in the app. Working on fixing this now…
Update, 12:44PM: This has been sorted out. We’re continuing to investigate the root cause and why our monitoring didn’t catch it sooner, but Blogger is back to functioning normally. Posts made from now on will be saved.
I’m very sorry to say that, if your blog was on this database, posts and template changes made in the last 18 hours or so were not saved. They may appear on your blog now, but will disappear if you republish. If you made a post between Friday afternoon and now, we suggest that you look at your list of posts (“Posting” tab, “Edit posts” sub-tab) and compare it with what is published on your blog. If posts are missing, copy them from your blog pages before you republish.
Major apologies to those of you who were affected. We work hard to maintain Blogger as a trustworthy place to keep your writing, and we really hate to let you down.
Update, 12:44PM: This has been sorted out. We’re continuing to investigate the root cause and why our monitoring didn’t catch it sooner, but Blogger is back to functioning normally. Posts made from now on will be saved.
I’m very sorry to say that, if your blog was on this database, posts and template changes made in the last 18 hours or so were not saved. They may appear on your blog now, but will disappear if you republish. If you made a post between Friday afternoon and now, we suggest that you look at your list of posts (“Posting” tab, “Edit posts” sub-tab) and compare it with what is published on your blog. If posts are missing, copy them from your blog pages before you republish.
Major apologies to those of you who were affected. We work hard to maintain Blogger as a trustworthy place to keep your writing, and we really hate to let you down.
Thursday, February 16, 2006
Thursday, February 09, 2006
Monday, February 06, 2006
The domain association problem mentioned yesterday has now been resolved. If you are still experiencing a problem with your mapped domain, please let Support know.
A reminder that we will also be down for planned maintenance starting at 7p (PST) this evening. We will be using this time to replace the older hardware responsible for many of the problems we experienced last week.
A reminder that we will also be down for planned maintenance starting at 7p (PST) this evening. We will be using this time to replace the older hardware responsible for many of the problems we experienced last week.
Sunday, February 05, 2006
Just a quick reminder that we will be going ahead with a planned network maintenance on Monday the 6th from 7–8PM PST. Blogger and Blog*Spot blogs will be unavailable during that time.
This maintenance won’t fix everything, but it will make things better. I promise.
This maintenance won’t fix everything, but it will make things better. I promise.
Saturday, February 04, 2006
As many people will have noticed, we've had significant hardware trouble over the past couple days. We have tried to keep all of you informed about the status of the service and our efforts to correct the problems. Unfortunately, today's problems were widespread enough such that we were unable to provide ongoing updates. One of the things we'll be doing in the short term is to come up with ways to prevent this.
Eric details the chronology below. There is one residual issue related to the outage: some legacy users have domain associations between Blog*Spot and external domains. These associations are not functional at this time.
In addition to working on this problem, we'll also be making the necessary changes to guard against the outages we experienced over the past 48 hours.
We will provide updates on our progress as changes are made. I sincerely apologize for the downtime - in the past six months I feel we've made significant strides to limit this type of failure. We obviously have more work to do and carrying it out will be our highest priority.
Eric details the chronology below. There is one residual issue related to the outage: some legacy users have domain associations between Blog*Spot and external domains. These associations are not functional at this time.
In addition to working on this problem, we'll also be making the necessary changes to guard against the outages we experienced over the past 48 hours.
We will provide updates on our progress as changes are made. I sincerely apologize for the downtime - in the past six months I feel we've made significant strides to limit this type of failure. We obviously have more work to do and carrying it out will be our highest priority.
Blogspot is again experiencing problems - we are investigating.
Update, 8PM: We have restored all of Blog*Spot, save one of our filers. This means that some blogs will still be unpublishable and inaccessible. Our engineers are continuing to work on this problem.
Update, 11PM: Blog*Spot servers are restarting now and connecting with the filer. All blogs should be publishable and accessible within the next 20–30 minutes.
[So how has everyone else’s weekend been? —Pete]
Update, 8PM: We have restored all of Blog*Spot, save one of our filers. This means that some blogs will still be unpublishable and inaccessible. Our engineers are continuing to work on this problem.
Update, 11PM: Blog*Spot servers are restarting now and connecting with the filer. All blogs should be publishable and accessible within the next 20–30 minutes.
[So how has everyone else’s weekend been? —Pete]
Friday, February 03, 2006
We are currently experiencing some problems with Blog*Spot. A percentage of blogs are completely inaccessible. For others, you may need to reload a few times before you connect to a web server that will serve them. We anticipate things will be calmed down within the hour.
Update, 5:20PM: Blogger and Blog*Spot should be fully operational with the exception of one of our file servers. This means a percentage of blogs will be unpublishable and inaccessible. We are currently waiting for a lengthy disk integrity check to complete on the file server. There may also be transient delays serving blog pages while our load balancing evens out.
Update, 8:00PM: The Blog*Spot web servers are coming back online with connections to the file server that was out for most of the evening. (Its disk is totally okay, btw.) So, within the next 10–15 minutes everything should be completely back to normal.
Again, super major apologies for this.
Update, 5:20PM: Blogger and Blog*Spot should be fully operational with the exception of one of our file servers. This means a percentage of blogs will be unpublishable and inaccessible. We are currently waiting for a lengthy disk integrity check to complete on the file server. There may also be transient delays serving blog pages while our load balancing evens out.
Update, 8:00PM: The Blog*Spot web servers are coming back online with connections to the file server that was out for most of the evening. (Its disk is totally okay, btw.) So, within the next 10–15 minutes everything should be completely back to normal.
Again, super major apologies for this.
Blogger.com is now back from a 45 minute outage. Two outages in one night. I feel like Dante from Clerks: “I’m not even supposed to be carrying the pager today!”
We’re really sorry about this spate of random downtime. As Jason mentioned, I mentioned that Monday’s fixup will lessen these types of outages in the future. The Blogger and Google engineers and ops folks are not just sitting around waiting for the next failure; we’re actively improving our infrastructure to lessen both the planned and unplanned Blogger outages.
We’re really sorry about this spate of random downtime. As Jason mentioned, I mentioned that Monday’s fixup will lessen these types of outages in the future. The Blogger and Google engineers and ops folks are not just sitting around waiting for the next failure; we’re actively improving our infrastructure to lessen both the planned and unplanned Blogger outages.
Thursday, February 02, 2006
Wednesday, February 01, 2006
Okay, big lead time on this one. Lots of time to prepare yourself.
Blogger and Blog*Spot will be offline for approximately one hour from 7–8PM this coming Monday, February 6th. The little outage indicator will kick in Sunday night to remind you.
This time, we’re completing the long–prepared for cutover from an older, flakier piece of network equiptment to a shiny bright new one. (This is how they explain things to software guys like me.) That means fewer planned and “unplanned” outages in the future. Seriously. I wouldn’t be too surprised if the ops guys get all Office Space on the thing they’re taking out, given the trouble it’s caused.
Now get thee back to blogging.
Blogger and Blog*Spot will be offline for approximately one hour from 7–8PM this coming Monday, February 6th. The little outage indicator will kick in Sunday night to remind you.
This time, we’re completing the long–prepared for cutover from an older, flakier piece of network equiptment to a shiny bright new one. (This is how they explain things to software guys like me.) That means fewer planned and “unplanned” outages in the future. Seriously. I wouldn’t be too surprised if the ops guys get all Office Space on the thing they’re taking out, given the trouble it’s caused.
Now get thee back to blogging.
Thursday, January 26, 2006
Tuesday, January 24, 2006
We'll be taking Blogger down on Wednesday the 25th at 4pm PST to fix a bit of a switch that's gone wonky on us. The outage should last about 15 minutes. Blogger.com and Blog*Spot blogs will be inaccessible during this time.
This repair will fix the problem that caused the brief outage last Friday night. We're also using this down time as an opportunity to tune our databases for more efficient spam catching and deletion.
Update:
(P.S.: The switch and the rest of Blogger is cheerful and productive.)
This repair will fix the problem that caused the brief outage last Friday night. We're also using this down time as an opportunity to tune our databases for more efficient spam catching and deletion.
Update:
Theres a notice saying "scheduled outage at 4:00 PST" on blogger. I spent the whole time of writing this trying to figure out what they might do to cause a scheduled outrage before I noticed the lack of an r.Found on McBlog
(P.S.: The switch and the rest of Blogger is cheerful and productive.)
Monday, January 16, 2006
Over the weekend, some users were seeing generic "unexpected error" messages when attempting to log in to their accounts. That problem has been patched as of about 10 AM this morning (PST), so if you were affected by it your account should be working normally again now. If you still see the error, just clear your browser's cookies, then log in again.
At 6am, some Blogger users suffered a 10-minute outage while a database took a nap. We apologize for the inconvenience while we continue to improve our service.
Update: After some early morning investigation, we further pinned down and isolated the issue. We will be taking steps this week to ensure this particular problem does not occur again.
Update: After some early morning investigation, we further pinned down and isolated the issue. We will be taking steps this week to ensure this particular problem does not occur again.
Thursday, January 12, 2006
Subscribe to:
Posts (Atom)