Page 1 of 1

*Resolved* Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 12:29 pm
by Clewby
Not knowing how to raise this in any other way, I'm leaving this message.

When attempting to access the Spamgourmet website, I get the following (timed at 2013-05-20 12:28 GMT):

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, [no address given] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
Apache/2.2.9 (Debian) mod_ssl/2.2.9 OpenSSL/0.9.8g mod_perl/2.0.4 Perl/v5.10.0 Server at spamgourmet.com Port 80


Also, the flow of forwarding appears to have stopped. Is there a major problem hitting Spamgourmet at the moment?

Oh, and just to add, http://www.isitdownrightnow.com/spamgourmet.com.html says that the site has been down for just over 23 hours.

Additionally, Sami Lehtinen's status checker ( http://s.sami-lehtinen.net/c/sc?sg ) shows:

Status: ERROR!



Server is not reachable!

Details:
CONNECT: [timeout / connection lost before this check point]
HELO: [timeout / connection lost before this check point]
QUIT: [timeout / connection lost before this check point]

Simple SMTP monitoring Python 3 script by: Sami Lehtinen

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 3:12 pm
by josh
There was a hard drive failure - we're working to get the service back up. I'll post updates here.

Josh

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 4:07 pm
by Clewby
Thank-you for the update, josh. I hope the restore goes smoothly without further headaches for you!

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 4:13 pm
by lwc
It's still like this. I'm missing important incoming as well as outgoing (reply masked) messages! Is this related? Will those messages arrive eventually?

Actually, it has been like this even before today.

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 5:36 pm
by Clewby
lwc wrote:It's still like this. I'm missing important incoming as well as outgoing (reply masked) messages! Is this related? Will those messages arrive eventually?

Actually, it has been like this even before today.


In most cases the messages will arrive eventually. Most Message Transfer Agents (MTAs) are configured to retry a certain number of times before they give up. This means that messages sent to a Spamgourmet address will generally be queued by the last MTA in the chain before Spamgourmet, waiting for Spamgourmet to become available again. If the outage is a very long time*, some systems will report to the sender that the mail was undeliverable.

In the case of messages sent out, your mail client will keep trying until it succeeds, or report back that the mail was undeliverable.

Be aware that some systems give up at the first attempt.

*'A very long time' is variable. It depends on what the person setting up the MTA decided was long enough to keep retrying.

Here is an example of a retry interval policy. Policies vary.

If a mail cannot be delivered at the first attempt, the MTA tries to resend the message every 15 minutes for 2 hours. Then the retry intervals are increased, starting at 1 hour and increasing each time by a factor of 1.5, up to 16 hours. Then the MTA retries every 8 hours until 4 days have passed since the first failed delivery.

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 5:55 pm
by josh
The service provider notified me pretty much right away due to monitoring. We're still not back up, but it's highly unlikely that we'll be down for a Very Long Time. Thanks for your patience everyone. I'll keep posting.

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 8:12 pm
by josh
checking in - we've had hard drive failures in the past. Normally we start with a fresh install, then I get the service running and configured and restore everything from backups. That works great when the hard drive fails on a weekend, which actually hasn't happened. Normally it happens right as I'm about to board a trans-atlantic flight. Anyway, today I'm tied up for most of the day with $DAYJOB, and so the service provider (cari.net, who rocks, btw) is imaging the damaged hard drive to a new one. That takes more time than slapping in a new clean server image, but given my current situation I believe it's going to be the quickest path to getting back in service.

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 9:36 pm
by josh
Still waiting for the hard drive imaging process to complete.

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 9:52 pm
by josh
cloned drive is going back in the server now...

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 10:16 pm
by dan78
The home screen is now loading! Yay!!! Thank you for running this website along with your real day job! I'm sure it's not easy and mostly thankless, but I depend on it every day...

-Dan

Re: Spamgourmet website apparently offline

PostPosted: Mon May 20, 2013 10:27 pm
by josh
The service is back up - I'm still poking around to make sure everything's OK. So far, so good.

Note that the server will be slammed for a while as it catches up.

Re: Spamgourmet website apparently offline

PostPosted: Tue May 21, 2013 2:41 am
by jimlaur
I did not realize how much I needed spamgourment until I thought I might have lost it. It gave me quite a scare. I am paying more attention now. I am indeed very grateful to you for creating this service.

Re: Spamgourmet website apparently offline

PostPosted: Tue May 21, 2013 7:39 am
by Clewby
Thank-you for bringing Spamgourmet back to life, josh.

Is there anything we can do to make situations like this less traumatic? Would an infusion of capital to buy an extra hard drive and/or an SSD help? I'm sure there are plenty of people who value Spamgourmet who would be willing to chip in.

Re: *Resolved* Spamgourmet website apparently offline

PostPosted: Tue May 21, 2013 8:11 am
by Clewby
Just to say I have donated by buying something off SysKoll's Amazon wishlist (https://www.amazon.com/gp/registry/wish ... GI2M4372Z9). I'm not sure if that keeps josh happy... :-)

If anyone feels like donating, the PayPal link is here, also: http://www.spamgourmet.com/index.pl?pri ... onate.html

Note that, as the page says, "Donation is not required. "

Re: *Resolved* Spamgourmet website apparently offline

PostPosted: Tue May 21, 2013 2:45 pm
by josh
Thanks everybody!

The service provider is offering a beefier box with RAID, which would likely prevent the problem we had yesterday. It's a little more expensive, but not too much. I'm looking into it.

Josh