service outage

Discussion of items in the "What's New" log.

Postby info » Tue Dec 14, 2004 7:30 pm

the server is still catching back up. The combination of the backlog created by downtime and the recent scripting by some sg users (resulting in a huge disparity of resource consumption) have got it under the gun. Mail *is* coming through, but there are thousands of messages in the outbound queue.

We're continuing to work to speed things up.
info
Site Admin
 
Posts: 100
Joined: Thu Aug 28, 2003 12:54 pm

Email

Postby Flash » Tue Dec 14, 2004 9:17 pm

Two emails that I was expecting on 13th and 14th December have not arrived. A test message which I sent on the 13th Dec, arrived within 2 hours ( usual time can be less than a minute ). The two emails may turn up yet, but I can live with it if they don't.
S.G.has been very reliable, in fact more reliable than the addresses which my emails are forwarded to. I have faith that all will be rectified shortly.
Thanks for a great service, and long may you continue !
Flash
 

Postby Guest » Wed Dec 15, 2004 11:02 am

I'd like to express some concern over this - I have received one email from SpamGourmet on the 13th and nothing since (sent a test email to myself on the 14th, I typically get 5-15 emails/day). If SpamGourmet is working though a backlog, in what order is it processing messages? (by account name, message datestamps, sending server, etc). How much more of a backlog is left for it to handle and is it responding to new incoming messages as it does so?

While I certainly appreciate the information given (and the service supplied so far), an outage of this magnitude is a real concern.
Guest
 

Postby basti (no login) » Wed Dec 15, 2004 11:03 am

I'm waiting for several messages from dec. 12th as well. I have received two of about 15 expected ones. At the moment, no bounce emails are coming back, so i guess email is going through but takes several days at the moment.

@info: when do you think the remaining outgoing emails will be completed?
basti (no login)
 

Postby Paranoid2000 » Wed Dec 15, 2004 11:05 am

BTW, that last Guest response was me - the system logged me out for some reason. :?
Paranoid2000
 
Posts: 71
Joined: Wed Dec 15, 2004 10:48 am

Postby Mstanlita » Wed Dec 15, 2004 1:37 pm

If SpamGourmet is working though a backlog, in what order is it processing messages? (by account name, message datestamps, sending server, etc). How much more of a backlog is left for it to handle and is it responding to new incoming messages as it does so?

While I certainly appreciate the information given (and the service supplied so far), an outage of this magnitude is a real concern.


This is a reasonable question. Should I be assuming the mail from the 11th on that I haven't got, I will never get and take it from there? I think an "official" response is overdue. This has been a mess and an embarassment for me as I am I'm sure for others.

But I still love you SG!! (As I say after reprimanding my daughter.)
Mstanlita
 
Posts: 3
Joined: Tue Dec 14, 2004 1:52 pm

Postby jbs » Wed Dec 15, 2004 2:11 pm

I think some others might agree that a status update would be incredibly useful here, even if the time to write it delays the arrival of the resolution slightly.

There's a temptation when putting out a massive fire to focus solely on the fire and not on communication, figuring you want to get it fixed ASAP. But for the thousands of users out there wondering whether the service will ever come back, they'd far prefer to hear a bit of news every day or so, just to manage their expectations.

It's hard to give feedback without sounding ungrateful, so I'm sorry if it comes across that way. I know how hard the guys running this service work to make it available, for free, and I appreciate that. I contribute financially what I can, and I've posted a poll about moving to a paid service b/c I think they deserve to be compensated.

But there's obviously an expectation by users of a certain availability, and when that expectation can't be met, a little communication goes a LONG way.

Thanks for all your hard work, SG guys, and if you get a chance, just let us know how it's going!

--Jason
jbs
 
Posts: 36
Joined: Fri Jun 18, 2004 4:51 pm

I agree

Postby DJA » Wed Dec 15, 2004 3:28 pm

jbs wrote:I think some others might agree that a status update would be incredibly useful here, even if the time to write it delays the arrival of the resolution slightly.

There's a temptation when putting out a massive fire to focus solely on the fire and not on communication, figuring you want to get it fixed ASAP. But for the thousands of users out there wondering whether the service will ever come back, they'd far prefer to hear a bit of news every day or so, just to manage their expectations.

It's hard to give feedback without sounding ungrateful, so I'm sorry if it comes across that way. I know how hard the guys running this service work to make it available, for free, and I appreciate that. I contribute financially what I can, and I've posted a poll about moving to a paid service b/c I think they deserve to be compensated.

But there's obviously an expectation by users of a certain availability, and when that expectation can't be met, a little communication goes a LONG way.

Thanks for all your hard work, SG guys, and if you get a chance, just let us know how it's going!

--Jason


I agree with Jason. Because this service is so useful and up to now, very reliable, many of us have taken this for granted. It's easy to bitch when there is a problem, but SG is provided as a free and non-warranted service. Anyone who is unhappy should simply look for an alternative without making a big deal about it.

Anyone who's administered anything knows problems will occur. Even fee-based services have outages. Stuff happens...

Having said that, if you are going to undertake something like this, you need to remember there are many people impacted by an outage and regular updates keep the community calm. If you think it will be a day, a week, a month, whatever, just let us know what's happening.

Good luck getting this sorted out and thanks for the hard work.

David
DJA
 
Posts: 9
Joined: Fri Mar 12, 2004 9:34 pm

Postby SysKoll » Wed Dec 15, 2004 4:52 pm

Status update: as of now, there are about 600 messages in the queue. This is almost normal for SG. So the catching up is almost done.
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm

Postby Paranoid2000 » Wed Dec 15, 2004 5:00 pm

So should those who have yet to receive a single email from SpamGourmet since 12/12 contact you directly to investigate their account?
Paranoid2000
 
Posts: 71
Joined: Wed Dec 15, 2004 10:48 am

Postby jbs » Wed Dec 15, 2004 8:42 pm

SysKoll wrote:Status update: as of now, there are about 600 messages in the queue. This is almost normal for SG. So the catching up is almost done.


That's definitely very bad news, since there are at least 50 messages which never showed up to me during that time. I've gotten about 8 or so in the last few hours, but unless I alone represent 1/12 of the queue, it appears a lot of messages must have been lost.

--Jason
jbs
 
Posts: 36
Joined: Fri Jun 18, 2004 4:51 pm

Something is still wrong

Postby JO » Wed Dec 15, 2004 10:07 pm

Aha, I think I found out something critical -- I think this is related to GMail.

My SG was set to forward to my gmail account. I had been receiving e-mails up to about the 12th, and nothing since.

So I sent a message to a new disposable. The new disposable appeared and showed that it had forwarded one message. The message did not appear in Gmail. So SG had received it, but not forwarded it. I sent a message to my GMail account and it appeared.

So I went in and changed my forwarding address to be my Yahoo address. I received the confirmation e-mail and confirmed my new e-mail address. Then I sent a message to the same disposable and it was delivered to my Yahoo address (message count changed to 2).

So then I went in and changed my forwarding address back to my gmail address. I got the confirmation email in my gmail account and confirmed the new address. Then I sent a message to my disposable and it never arrived in the gmail account. I checked SG and it never showed that it received another disposable (message count was still 2). So I tried again, same results.

Reconfigured again to have the mail delivered to my Yahoo account.

Then I configured it back to Yahoo, got the confirmation e-mail and confirmed the change, and sent another message to the disposable. It showed up (message count 3).

Here's what I think is going on. Apparently SG CAN send mail to GMail (because it could send the confirmation message), but it can't forward mail to GMail anymore.

Now here's something else that may play into this: I remember reading somewhere, and I can't find it anymore but I thought it was on GMail, that they were changing it to reject mails from improperly configured servers and that if you weren't getting the messages from somebody, you had to contact their ISP to get them to properly configure it. This memory is very vague, so I'm not sure if any of that exact. I know there was something about improperly configured servers, and I thought it was related to GMail. Don't know if any of this helps, but I'm back up and running again.
JO
 

oops

Postby JO » Wed Dec 15, 2004 10:12 pm

Sorry, that message was a little bit confusing. Just some clarifying points.

In the first part, when I "sent a message to my gmail account" I meant I sent it directly to the gmail account. This was to confirm that gmail was capable of receiving e-mail.

Paragraph 6 can be deleted -- I accidentally repeated twice about setting up the Yahoo account. I guess I better get registered so I can edit my messages ;)
JO
 

Postby info » Wed Dec 15, 2004 11:46 pm

Here's what's going on now and how it relates to delayed messages and the queues:

During the time that we were down completely, mail servers that wanted to send mail to spamgourmet couldn't, so presumably they queued it (some may have given up right away -- we can't know), and started to retry at some regular interval.

When we came back up, the server was, of course, hit with a huge number of connect requests -- more than it could handle. Those remote servers who made a connection got their mail into our server. Those who couldn't kept trying. As time has worn on, the number of requests is only now consistently falling below our ability to handle them. If you were following the news here, you'll know that some sg users had been scripting addresses, and so had caused a really high load -- this, along with our normal traffic, compounded while we were down, and no doubt continued to compound (albeit at a rate that was slower than our deliveries) as the server was maxed out and rejecting connections. For years, our ratio of eaten to delivered mail was about 10 to 1. Recently, it's
been more like 10 to 2.5, which is unhealthy for us at the volume we're handling. I'm not sure if this is because more users have figured out how to make addresses forward permanently (there's a reason for why we discouraged this :) ) or if it's something else, but the ratio has changed.

As of now, we're still spiking up to the point where we reject connections, but that percentage of the time has gotten small and will hopefully stop altogether soon. We had also taken some steps to reduce the effect of the scripted addresses, and this is helping too.

No doubt some mail servers have tried at all the wrong times and have given up trying to deliver messages. Others are likely still trying -- for these, messages are "en route" but not yet in our queue.

What can we learn from this? 1) basically that our service is pretty much at capacity, and that any downtime (and the resulting build-up in to-be-delivered mail) pushes it over the edge into a delay condition, and 2) we probably need to figure out how to restore our normal ratio of eaten to delivered messages -- this *may* have to involve a change to the exclusive sender logic.

Issue number one could probably be solved with better hardware, but as you all know we don't have the budget for that -- what we can do is refactor the software to run more efficiently. From here on, all our spamgourmet efforts will be directed at that.
info
Site Admin
 
Posts: 100
Joined: Thu Aug 28, 2003 12:54 pm

Postby Paranoid2000 » Thu Dec 16, 2004 12:17 am

info wrote:When we came back up, the server was, of course, hit with a huge number of connect requests -- more than it could handle. Those remote servers who made a connection got their mail into our server. Those who couldn't kept trying. As time has worn on, the number of requests is only now consistently falling below our ability to handle them.
Thanks for the information - this presumably does mean that previous emails may be lost if the sending server decided to "give up" on SpamGourmet, but at least the sender should be notified in such a case. I have sent a second test mail and it went through pretty much instantaneously.
info wrote:For years, our ratio of eaten to delivered mail was about 10 to 1. Recently, it's been more like 10 to 2.5, which is unhealthy for us at the volume we're handling. I'm not sure if this is because more users have figured out how to make addresses forward permanently (there's a reason for why we discouraged this :) ) or if it's something else, but the ratio has changed.
I can't speak for others, but one of the uses I make of SpamGourmet is for forum notification emails. These I want forwarded always, but I also need to have the ability to disable them in the event of someone on those forums finding (and abusing) those addresses. This may not be what SG was intended for, but for me it is a highly valued feature. Also if I ever change ISPs, switching to their email address simply means updating one SG setting rather than dozens of assorted websites (BTW my forwarded/eaten ratio is 10/4).
info wrote:2) we probably need to figure out how to restore our normal ratio of eaten to delivered messages
Send the SG database in a brown paper package to Alan Ralsky... :D
info wrote:Issue number one could probably be solved with better hardware, but as you all know we don't have the budget for that -- what we can do is refactor the software to run more efficiently. From here on, all our spamgourmet efforts will be directed at that.
This is really where offering a commercial option would help. Heavy users (either with lots of aliases or a high forward/eaten ratio) are likely to be willing to pay to regain SG's previous reliability (I certainly would and have voted in the poll). Also with ~80,000 accounts, SpamGourmet should be large enough to merit a backup server.
Paranoid2000
 
Posts: 71
Joined: Wed Dec 15, 2004 10:48 am

PreviousNext

Return to What's New

Who is online

Users browsing this forum: No registered users and 19 guests