Mail disappearing from OpinionJournal.com

[jbs]

I'm a devoted (read: paying ) Spamgourmet user and find the service indispensable. I have over 100 SG addresses to date, and have never had any trouble with any of them.

There's a newsletter I read, however, which I've never received through the SG address I gave them. It's from a site called www.opinionjournal.com. I always assumed that they were blocking the xoxy.net address for some reason, and I just ignored it.

Recently, though, I decided to look into it, and received a very thorough reply back from their sysadmin, the gist of which is they are sending it to my address (actually 2 SG addresses I've given them).

Neither have run out down their count, and neither shows ever having deleted a message. The sysadmin there sent me the log files from yesterday, showing delivery to my SG addresses, and some others, all of which I'll block out below:

Apr 12 16:49:42 djoj sendmail[8248]: QAA08246: to=<opinionjournal.20.********@xoxy.net>,<botw.20.********@xoxy.net>,<dowj.6
.******@xoxy.net>,<wsj.20.******@xoxy.net>,<botw.15.********@xoxy.net>,<wsj.t
o.****@xoxy.net>,
delay=00:00:51, xdelay=00:00:07, mailer=esmtp, relay=gourmet.spamgourmet.com. [216.218.230.146], stat=Sent (2.0.0 i3CKnfQ8019754 Message accepted for
delivery)

But I've never received any of the messages they've sent. In both cases, I should be receiving one mail a day, but the record on my SG status page shows:

opinionjournal 20 14 6 0 (max, remaining, sent, deleted)
botw 19 16 4 0 (max, remaining, sent, deleted)

The few sent correspond to the confirmation emails which I received after signing up, and I think in each case one or two messages came through before they stopped. In any case, since I've been signed up at each address for several months, the Sent count should be in the tens or hundreds by now. The fact that neither the Sent nor the Deleted is increasing tells me that this can't be an issue with the email service to which SG forwards, since they're never getting the chance to block it. In any case, that service is Runbox, and they don't do any of the AOL or Hotmail style black-hole deletions of spams without telling users.

Now that I know OpinionJournal is sending them, though, the only thing I can POSSIBLY imagine is that they are somehow being lost between the time SG receives them and sends them on.

Can anyone shed any light on this?

Thanks!!

--Jason

P.S. If it would be helpful to have my user name, I can email or PM it to Josh or an admin.

[josh]

I think this is what's happening -- it's definitely a spamgourmet thing:

When I first wrote the code years ago, I only considered that a message would be sent to one spamgourmet address (and this was indeed true for the longest time), and so the code only looks for one. The upstream mail server only sends one copy of the message to spamgourmet for all the spamgourmet addresses, because, as far as it knows, they're all going to the same place. Spamgourmet then picks one of the addresses and sends the message on its way (or not, as the case may be).

We identified this awhile back, and placed a low-ish priority on it because, at the time, no one had come forth with a problem because of it, and it unintentionally serves as a defense against blanket spamming of spamgourmet accounts, for obvious reasons.

I'm pretty sure that's what's going on -- not the best news I could give, I know. There may be a temporary work-around for you. I noticed that all the accounts in that list used xoxy.net, so perhaps if you signed up with a different domain name (dfgh.net?), you'd be the only one with it and their system would send a separate message to you.

Meanwhile on our end, I'll escalate the issue in development, and hopefully we can figure out how to fix the problem and somehow preserve the defense I mentioned. I'll also look for ways to cause the upstream mail server to split the message for multiple recipients, which would require no changes to sg code, but would defeat the defense -- if that can be done, perhaps we can set things up that way while we work on the code.

[Guest]

We identified this awhile back, and placed a low-ish priority on it because, at the time, no one had come forth with a problem because of it, and it unintentionally serves as a defense against blanket spamming of spamgourmet accounts, for obvious reasons.

I'm pretty sure that's what's going on -- not the best news I could give, I know. There may be a temporary work-around for you. I noticed that all the accounts in that list used xoxy.net, so perhaps if you signed up with a different domain name (dfgh.net?), you'd be the only one with it and their system would send a separate message to you.

So first off it's nice to know that I'm not going nuts, but I'm actually pretty surprised this is the first time it's come up -- wouldn't any two (or more) people signing up for a group mailing experience this same phenomenon? I would imagine it's very common for a legitimate opt-in mass mailer to bundle their messages to a given domain, isn't it? I wonder if this is happening with other people but, like me, it's always been a low level enough annoyance that I didn't look into it.

I'm also sorry to report it may be even worse than you think -- I just did a quick test sending from myself to an xoxy address of mine and cc'ing an xoxy address of my wife, and NEITHER ONE was delivered. I immediately followed this with individual messages to the same 2 addresses and both came through fine. So it appears that the SG mailserver is not picking one of the addresses for delivery, but may in fact be ignoring them entirely.

I did change my address to another domain, but keep in mind the only reason they all appear to be using xoxy.net is that that is the only section of the log file he sent to me. He was not looking for addresses at dfgh.net, so they may also have subscribers there, but there would be no reason for him to include those in his email to me.

I'm going to share the response with the folks at OpinionJournal, but it seems unlikely that they would be willing to break up the messages into individual ones, since they're following legitimate mail protocols. I suppose there's an argument that a bulk distribution is more likely to be falsely declared spam by any number of server-side anti-spam regimens than would individual messages, but still . . .

Josh, thanks for figuring this out. I'll continue to put thought into other workarounds, or fixes that would preserve the defense. One slightly weaker defense might be to look for blanket spams where the UniqueID and Count were the same, for example this:

BobsList.20.john@
BobsList.20.sue@
BobsList.20.ann@

looks more suspicious to me than:

OpinionJournal.20.john@
DailyNews.15.sue@
Articles.12.ann@

but that would be pretty easy to defeat as well . . . perhaps only honoring group sends to multiple recipients when the sender is on a trusted sender list? As I think about it, though, I can think of more and more situations where this might come up -- even something as simple as someone writing to me and cc'ing my wife (who also uses xoxy) means neither one of us would receive it.

I'd certainly be in favor of delivering all messages rather than none. Watchwords are always available to any user who experiences blanket spamming, but the way it works right now messages are going undelivered which the user never knows about. I never realized this was a possibility, but with all the services I've signed up for using SG addresses, I can't believe it's not happening elsewhere -- Amazon, iTunes, my alumni newsletter. The more popular SpamGourmet becomes, the more often this is going to come up, and people won't find out about it until they miss an important message . . .

Thanks again for getting to the bottom of this. Would you mind posting back when you think this might change? Thanks!!

--Jason

[Clyde]

I was quite surprised to read about this problem here--and nowhere else on the SpamGourmet website.

Is it really true that if 2+ SG users ever sign up for a commercial mailing list or similar service (with the same SG domain name), and that mailing list or similar service addresses email to multiple recipients grouped by domain name for efficiency, none of the SG recipients will receive the message?

If so, it severely limits SG's usefulness, and could even be perceived as a deceptive practice. After all, the most likely uses of SG involve email sources that send to multiple recipients.

IMHO, the fact that "it unintentionally serves as a defense against blanket spamming of spamgourmet accounts" is not a sufficient justification to allow this to continue.

I'm not an SMTP expert, but it seems to me that to be RFC-compliant, once it accepts a connection request an SMTP server should process ALL incoming recipient mailbox addresses properly, even when the "proper" result may ultimately be to discard the message without delivering it to anyone.

I really think you should update the SG website to warn new and existing users about this problem.

[josh]

deceptive? Please refer to the first question in the FAQ that says we eat all the email, and the rest that talk about how you shouldn't use it for anything important, and the many others that attempt to communicate that thought -- spamgourmet is intentionally *not* SMTP compliant, by the way -- that's what dispsosable email is all about.

It's important to note that the issue does not affect mail from many lists -- there are several I know to be working with multiple sg subscribers (because I'm one of them), and this is the first we've heard of the problem in over three years. Previously, when we've received support requests from people having trouble with mailing lists, the problem has been something else.

That said, we're working pretty hard to resolve the problem, and I'll put something up on the what's new page.

[jbs]

Thought I'd check back in on this issue and see if anything had changed with regards to multiple SG users receiving the same message at the same domain. I just did the same test I mentioned above, sending an email to my xoxy account and my wife's account simultaneously, then sending the same message separately to each account.

The 2-recipient message disappeared, did not go to either one of us, while the separate recipient message came through fine.

Any plans to change this? I know there are pros and cons to the way it's currently handled, but I for one would definitely prefer to get the messages.

--Jason

[josh]

We're still working on it -- it's our top priority. Because of the high risk involved in changing the way we handle recipients, we're spending some time on it.