System Status: “body of message not found” issue

January 13th, 2009 by Tom Alison · comments · http://blog.zenbe.com/ycylx

Update (1/15 10:42am Eastern):

We’re currently restoring messages from secondary storage. Some metadata regarding the read/unread state and tags may be lost in the restore. So you may notice your Inbox counts go up in response to older messages being restored. The restore will happen over the course of the day and you should be able to access your content again by this afternoon.

Update (1/14 1:30pm Eastern):

Our IMAP server has been re-enabled. All messages not affected by the situation described below are available via IMAP.

Update (1/14 11:00am Eastern):

Our data center technicians discovered that one of primary file storage nodes has failed. This storage node held email data from POP accounts that we retrieved between 12/29 and 1/12. Any email addressed to your zenbe.com account was not affected.

We’ve been working continuously to try to recover the data from the primary node, but we are now shifting gears to restore the email content from our secondary backup archival node. Unfortunately this could take us several hours today. We will provide timely updates as to our progress in this blog post.

As mentioned previously, because this problem only affects POP messages from non-Zenbe accounts, as a workaround you can login to your external account to view the message. Zenbe never deletes messages from your external email accounts so you should still have a copy there.

Yesterday afternoon we took down our IMAP servers as they were behaving strangely in response to the missing files. We are working on restoring IMAP this morning. Some IMAP clients may behave strangely, but we are performing testing so we can let you know what to expect.

Again, we apologize for the inconvenience. We design all of our storage systems to be fault-tolerant. Unfortunately we experienced an abnormal malfunction of the storage hardware that we are working with the data center technicians to diagnose.

All POP email received after January 12th should be functioning normally.

This entry was posted on Tuesday, January 13th, 2009 at 9:54 am and is filed under Maintenance. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

14 Responses to “System Status: “body of message not found” issue”


  1. M Kennedy says:
    January 13, 2009 at 11:35 am

    It’s good to see timely warnings to users, of server problems, thanks and keep it up.

  2. G Griffin says:
    January 13, 2009 at 5:57 pm

    I agree! It’s this kind of service that adds the icing to a beautiful interface. Nice.

  3. Sean says:
    January 13, 2009 at 7:17 pm

    Any status on IMAP being re-enabled?

    Thanks

  4. mikedk says:
    January 14, 2009 at 4:51 am

    08.50 GMT. Are we not due an update?

  5. KPRussell says:
    January 14, 2009 at 11:44 am

    Is there an update on the outage? I’m still getting “body of message not found” error at 10:43 A.M. EST

    Thanks for your help.

  6. Tom Alison says:
    January 14, 2009 at 12:26 pm

    Hi everyone,

    Sorry for the gap in status updates. We’ve been working hard to recover the data from a corrupted storage node. I just updated the original blog post to give you the latest status.

    In a nutshell, one of our primary storage nodes that held POP email from external accounts collected over the past two weeks failed in an abnormal way. We worked with our data center technicians to try to recover the data from the primary node, but are now working on restoring from the secondary backup node. This process may take some time (we’re estimating 1-2 days at worst) but we will restore the email content.

    We’re bringing up the IMAP server this morning. Some IMAP clients may behave oddly when trying to retrieve messages that were affected by the outage. We’re working to identify which clients may behave abnormally and will let you know.

    As I mention in the blog post, only your external email account emails retrieved between 12/29 and 1/12 were affected. None of your zenbe.com email was affected. Because we never delete email from your external accounts, as a temporary workaround you can still login to your non-Zenbe email provider to view the message there.

    Again we apologize for the inconvenience and will keep you updated as we make progress on the restoration.

  7. Carter says:
    January 14, 2009 at 2:39 pm

    Only webmail provider I know of that interacts with it’s user population and as “M Kennedy” says “It’s good to see timely warnings to users, of server problems”. This is vital to users to be keep up to date and Zenbe day in day out comes through.

    Thanks again to the Zenbe team for all their hard work !!

    **Now if this forum allowed for me to add a gif you would see BIG thumbs up :-)

    .

  8. Alexander P. says:
    January 14, 2009 at 9:09 pm

    My zenbe account does not work on my laptop anymore and will also not work with imap on my iphone!

    Help!

  9. waseem says:
    January 15, 2009 at 1:14 am

    i am not been able to attach a file with email. tried so may ways to get it done but have not been bale to get through… someone could help me about it.
    thanks in advance,,

    waseem

  10. Jeremy W says:
    January 15, 2009 at 4:14 am

    I think this is affecting more than mails “that we retrieved between 12/29 and 1/12″. I am finding that I cannot open any emails that I received from 1st December 2008 onwards until 12th January 2009 inclusive.

    The last update that we’ve had was at 1.30pm Eastern time yesterday – I was hoping that we would have had another update on this by now. I guess we will have to wait until the US gets in before we hear anything back.

    I understand that this is a major issue and that it is rare to get ANY kind of feedback from a company in this situation, but for those of us who rely on Zenbe as our one-stop-shop solution, it is very frustrating when we are having to look in multiple places to deal with our emails depending on when we received them.

    That said, I am still very grateful to all at Zenbe for keeping us informed and hope that we can get this sorted out as soon as possible.

  11. Tom Alison says:
    January 15, 2009 at 11:46 am

    We’ve begun the restore from secondary storage. The reason this took so long is that we were trying to preserve metadata like read/unread state as well as tags, but it looks like we’ll be unable to do that.

    As the restore proceeds over the course of the next few hours you may notice your unread counts go up in response to the messages being restored.

  12. Bravenue says:
    January 16, 2009 at 6:29 am

    Looks like everything is back to normal. In my account at least the body-less mails can be viewed now, but appear as unread (which of course is no problem). Thanks!

  13. Brenton says:
    January 20, 2009 at 2:56 pm

    Zenbe has sucked pretty hard lately on WebKit browsers. I doubt this is a related issue, but they seem to be coincident.

    I can never get back to my Inbox. Clicking Inbox gives me the “Loading Inbox” spinning flower – sending a message gives me a “sending” flower. Occasionally, I can click to Favorites and then Inbox as a workaround. Also, the up/down arrows are disabled (though they’ve always been hit/miss for me).

    I’ve never had a reason to try gMail before, but if this keeps up I may have to defect. . .

  14. Leif Skr says:
    February 6, 2009 at 5:32 am

    Even today the 5th February there are still only a message telling me “Loading Inbox”.
    What is happening?? Nothing or??