sg and spamcop

Discussion re sg development. You don't have to be a developer.

sg and spamcop

Postby josh » Thu Oct 16, 2003 2:23 pm

As we know, sg was blacklisted at spamcop based on spam reportedly originating from the sg server.

I've looked at this a bit and I don't believe this is purposeful on anybody's part -- an example of the reported headers are:

From x Thu Oct 16 01:05:42 2003
Return-Path: <jqh1@gourmet.spamgourmet.com>
Received: from gourmet.spamgourmet.com ([216.218.230.146])
by mail1.aus1.texas.net (8.11.6p3/8.11.6p3) with ESMTP id h9GAqwi11511
for <x>; Thu, 16 Oct 2003 05:52:58 -0500 (CDT)
Received: from localhost.localdomain (localhost [127.0.0.1])
by localhost (8.12.10/8.12.9) with ESMTP id h9GB3wPK004142
for <x>; Thu, 16 Oct 2003 04:03:58 -0700
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0
X-Mailer: MIME-tools 5.41 (Entity 5.404)
Date: Thu, 16 Oct 2003 03:52:55 -0700 (PDT)
From: x dan <alhaji@ratrunner.com>
To: x
Subject: urgent reply (satxrr: message 2 of 6)
Reply-To: x
X-Originating-Ip: [81.91.227.9]
Message-Id: <2003________________397A@sitemail.everyone.net>
Sender: ",,," <jqh1@gourmet.spamgourmet.com>
X-UIDL: I@H!!~#l"!BLY!!)"="!


I think this was automatically reported to spamcop via spamassassin or something similar, and this has likely been going on for sg's entire history. The difference here is that the headers *do* indicate that the message originated at the sg server, even though it's obvious to all of us that it didn't.

I *think* this caused by the Mail::Audit code discarding the old headers and "re-originating" the message when it sends. The old code preserved the previous headers and sg would correctly not appear as the originating server, and so never got blacklisted.

I have (with more than a little chagrin) once again reverted to the pre-Mail::Audit version of the code and I'll make some effort to prove up this theory in the next couple of days.

Syskoll, you reported the situation to Julian, no? I haven't contacted them yet, although I did join the mailing list. Do you think we should contact Julian again immediately or wait until we substantiate or disprove the theory above?

Josh
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm

Postby josh » Thu Oct 16, 2003 3:15 pm

I posted a quick announcement to news.admin.net.net-abuse.email, and I sent this to spamcop:

Julian (or blacklist admin),
The mail server at spamgourmet.com currently appears on the spamcop
blacklist, which is preventing a large number of our users from receiving
mail through unexpired disposable email addresses

You may be aware that spamgourmet is a disposable email address service
used primarily for the purpose of avoiding spam. It does not filter the
content of messages passing through the system, but rather relies on an
expiration system based on the number of messages that have been sent to a
particular disposable address. For this reason, mail that is considered
spam by any definition will pass through the site and to the user if sent
to an unexpired disposable address. That is, messages orginate from
anywhere, go to the spamgourmet server, and then are maybe (10% chance,
empirically) forwarded to the user's delivery address.

I believe that several such messages have been automatically reported to
spamcop as spam, and that spamcop believes these messages originated at
the spamgourmet server, when in fact they did not. I *believe* this is
due to our recent switch to perl mail libraries of Mail::Audit, which
might be "re-originating" the messages when they are sent, rather than
preserving the original headers and adding to them, as our older
home-grown code did. This is just a theory, I haven't had time to verify
it -- like you, I'm a volunteer with a day job. We'll drill down to
what's going on soon. In any event, I can assure you that we've seen no
reason to believe that spam messages are actually originating at the
spamgourmet server, and I'm sure you'll agree if you make the
slightest inquiry.

Anyway, for the sake of the many affected spamgourmet users, I'm asking to
be prematurely de-listed. If my theory above is right, I know it'll
happen anyway in 48 hours (we've temporarily switched back to the older
code), but that time could mean a lot of undelivered mail for the users.


Thanks,
Josh Hamilton
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm

That did the trick - SG delisted

Postby SysKoll » Thu Oct 16, 2003 5:56 pm

That seemed to work. SG has been delisted. The latest sample of "message reported as spam" on http://www.spamcop.net/w3m?action=check ... 18.230.146 now shows a report from a German SG user named snubbelfoot, yet we're now delisted:


Return-path: <jqh1@gourmet.spamgourmet.com>
Envelope-to: x
Delivery-date: Thu, 16 Oct 2003 15:43:04 +0200
Received: from [216.218.230.146] (helo=gourmet.spamgourmet.com)
by mxng14.kundenserver.de with esmtp (Exim 3.35 #1)
id 1AA8P5-0000XZ-00
for x; Thu, 16 Oct 2003 15:42:59 +0200
Received: from localhost.localdomain (localhost [127.0.0.1])
by localhost (8.12.10/8.12.9) with ESMTP id h9GDs0PK011051
for <x>; Thu, 16 Oct 2003 06:54:00 -0700
Message-ID: <0cce______________________fea9@noeske>
Reply-To: "Claudia - Claudia_Ke@freenet.de"<+lange+snubbelfoot+660c52323f.Claudia_Ke#freenet.de@spamgourmet.com>
From: "Claudia - Claudia_Ke@freenet.de"<+lange+snubbelfoot+660c52323f.Claudia_Ke#freenet.de@spamgourmet.com>
To: <x>
Subject: (lange: message 1 of 20)
Date: Thu, 16 Oct 2003 15:43:02 +0200
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0CC9_01C393FC.2ECE2790"
X-Priority: 3
X-MSMail-Priority: Normal


X-Mailer: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Sender: "Claudia - Claudia_Ke@freenet.de"<+lange+snubbelfoot+660c52323f.Claudia_Ke#freenet.de@spamgourmet.com>
X-Sender: Claudia - Claudia_Ke@freenet.de<+lange+snubbelfoot+660c52323f.Claudia_Ke#freenet.de@spamgourmet.com>
X-Sent-From: Claudia - Claudia_Ke@freenet.de<+lange+snubbelfoot+660c52323f.Claudia_Ke#freenet.de@spamgourmet.com>
Status: R
X-Status: N
X-KMail-EncryptionState:
X-KMail-SignatureState:

I believe you were using Mail::Audit.resend(). Does it really remove the Received headers? If so, that's really bad. We need to either find a way to preserve the Received headers or reexamine the rationale for using Mail::Audit. Amit?
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm

Postby josh » Thu Oct 16, 2003 7:17 pm

We're definitely using resend() -- remember that this is still just a theory :)

I'm slammed at work today, but I hope to be able to research it more closely very soon.
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm

Postby josh » Fri Oct 17, 2003 2:17 am

OK, spamcop agrees about the headers, and I tested it by sending messages from fastmail:


------------- with commandline sendmail ----------------------

Code: Select all
From XXXXXXXX  Thu Oct 16 16:59:37 2003
Return-Path: <jqh1@gourmet.spamgourmet.com>
Received: from gourmet.spamgourmet.com (localhost [127.0.0.1])
        by localhost (8.12.10/8.12.9) with ESMTP id h9GNxYPK011495
        for <XXXXXXX>; Thu, 16 Oct 2003 16:59:34 -0700
Received: (from jqh1@localhost)
        by gourmet.spamgourmet.com (8.12.10/8.12.10/Submit) id h9GNxY4N011493
        for XXXXXXX; Thu, 16 Oct 2003 16:59:34 -0700
Received: from out2.smtp.messagingengine.com (out2.smtp.messagingengine.com [66.111.4.26])
        by gourmet.spamgourmet.com (8.12.10/8.12.9) with ESMTP id h9GNxYPK011489
        for <testheaders.3.josh@spamgourmet.com>; Thu, 16 Oct 2003 16:59:34 -0700
Received: from www.fastmail.fm (localhost [127.0.0.1])
        by localhost.localdomain (Postfix) with ESMTP id D75F62FA7B0
        for <testheaders.3.josh@spamgourmet.com>; Thu, 16 Oct 2003 19:48:24 -0400 (EDT)
Received: from 10.202.2.132 ([10.202.2.132] helo=www.fastmail.fm) by messagingengine.com
  with SMTP; Thu, 16 Oct 2003 19:48:24 -0400
Received: by www.fastmail.fm (Postfix, from userid 99)
        id C77883ED73; Thu, 16 Oct 2003 19:48:23 -0400 (EDT)
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="ISO-8859-1"
MIME-Version: 1.0
X-Mailer: MIME::Lite 1.2  (F2.71; T1.001; A1.51; B2.12; Q2.03)
From: "josh h  - XXXXXXX"<+testheaders+josh+2292b87189.antichef#fastmail.fm@spamgourmet.com>
To: testheaders.3.josh@spamgourmet.com
Date: Thu, 16 Oct 2003 15:48:23 -0800
X-Epoch: 1066348104
X-Sasl-enc: ZCytTcn1C+V1rUtVVlRlog
Subject: test headers (testheaders: message 1 of 3)



-------------- with Mail::Audit -----------------------------


Code: Select all
From XXXXXXX  Thu Oct 16 17:00:07 2003
Return-Path: <sgtest@gourmet.spamgourmet.com>
Received: from localhost.localdomain (localhost [127.0.0.1])
        by localhost (8.12.10/8.12.9) with ESMTP id h9H005PK011624
        for <XXXXXX>; Thu, 16 Oct 2003 17:00:05 -0700
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="ISO-8859-1"
MIME-Version: 1.0
X-Mailer: MIME::Lite 1.2  (F2.71; T1.001; A1.51; B2.12; Q2.03)
From: "josh h  - XXXXXXX"<+testheaders+josh+c83cb0cd1d.antichef#fastmail.fm@spamgourmet.com>
To: testheaders.3.josh@clickablemelvin.com
Date: Thu, 16 Oct 2003 15:48:53 -0800
X-Epoch: 1066348134
X-Sasl-enc: NdGcv4NjePlhyy2EMpN0Ow
Subject: test headers with Mail::Audit (testheaders: message 1 of 3)



So, shoot... It does look like we're missing the previous headers with the Mail::Audit->resend() method. I'll try and look at that code and figure out if there's a workaround or if we can suggest a fix to the author.
Last edited by josh on Wed Oct 22, 2003 2:08 pm, edited 1 time in total.
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm

Postby maratheamit » Sat Oct 18, 2003 3:50 am

How about using the pipe() delivery method? Not sure if that would work, but worth testing out. I will try that on the test server over the weekend.
maratheamit
 
Posts: 82
Joined: Fri Aug 29, 2003 2:35 pm

Pipe sending method

Postby SysKoll » Sun Oct 19, 2003 2:21 am

I don't know if the Received headers are still there when the message is fed to the pipe. I intensely dislike this solution because it forces us to fork yet another process per email. That's what we wanted to avoid with the demonized spameater.

I'm much more in favor of modifying Mail::Audit or whatever dependent module (Mail::Internet maybe?) is removing the Received headers.
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm

Postby maratheamit » Sun Oct 19, 2003 3:11 am

I agree with you Syskoll. It would just help to know if there is a stop-gap alternative available till we can modify the Mail::Audit code. But then, we could just as well run with the old code till that point.

BTW, I haven't been able to test the pipe() method because the web site for the test environment is not up as yet.
maratheamit
 
Posts: 82
Joined: Fri Aug 29, 2003 2:35 pm

Postby maratheamit » Sun Oct 19, 2003 3:27 am

OK. The problem is in the _prephdr subroutine in the Mail::Internet module. There is a line at the begining
$hdr->delete('Received')
which is removing the old headers.

So we can just comment out that line for now. But long term it would be good to have this change in the next release of Mail::Internet. What is the best course of action. Should I email the author and suggest a config parameter for that behaviour?
maratheamit
 
Posts: 82
Joined: Fri Aug 29, 2003 2:35 pm

Postby SysKoll » Sun Oct 19, 2003 3:54 pm

Amit,

I second that. Remove the offending line in Mail::Internet, then email the author. I'd even go as far as sending him a proposed patch with the optional parm.
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm

Postby josh » Mon Oct 20, 2003 4:46 pm

sounds good - btw, it looks like we may have at least one header parsing issue with M::A, too:

Hi Josh,

Early Thurdsay morning saw blank subject lines. Everything since then (5 messages) has shown up with full subject lines, including "trusted sender for your account".

I will now go back and remove the trusted sender definition, to verify exclusive sender for address is working (I expect it will).

Once again, thank you very, very much!

Stephen


-- it appeared to be getting confused on that multi-line header message (somehow losing the subject line), and the problem went away when we switched back. Would it be worth including that in the communication as well? I haven't looked into it yet.

Beyond that, we'd probably do best to try for a new method in Mail::Audit (eg ->forward() ? or something) that preserves headers so that an upgrade wouldn't cause unexpected behavior in other existing implementations (all that is up to the authors, of course...) -- are the authors the same in Mail::Audit and Mail::Internet -- or are we only dealing with Mail::Internet as regards the ->resend() method and the header problem?
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm

What should we do? Modify our code or Mail::Audit?

Postby SysKoll » Mon Oct 20, 2003 6:42 pm

If I remember correctly, we decided to use Mail::Audit because of parsing problems in the spameater.pl code. But now we see other unexpected parsing problems in Mail::Audit.

So what should we do? Amit, Josh, what's the reasoning for using Mail::Audit? What's the advantage compared to reworking the spameater code?

We really need to evaluate the benefits of using Mail::Audit since we're headed towards butchering it.

This is is OT, I am opening a new thread. Please follow up there.
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm

Postby Guest » Tue Oct 21, 2003 9:52 pm

Hello.

I had started a thread relating to this and on a response from Syskoll, I changed from my regular email account that was being forwarded to, to a hotmail account to avoid the problem of the original email provider using a blacklist.

Based on the discussion here though, would it be correct for me to assume that it would be safe to switch back to my original email account that mail was being forwarded to since you have changed or reverted the code dealing with email headers?

Thanks,
Ben
Guest
 

I confirm

Postby SysKoll » Tue Oct 21, 2003 10:46 pm

Ben,

You can set your forwarding address back to its previous value. SG is not in the spamcop blacklist anymore so you shouldn't have any trouble.
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm


Return to Developers

Who is online

Users browsing this forum: No registered users and 15 guests