New disposible address shows sent count but no emails arrive

[josh]

arghhhh.. I went and signed up on gamersgate and tried to reproduce the issue - at first, it looked like I did - I received the confirmation message without masking, but didn't appear to receive it with masking. I was stoked. Then I saw the message with masking in my spam folder - gmail said " It seems to be a fake "bounce" reply to a message that you didn't actually send." - but the same was not true for the ones sent without masking. Sigh...

Which gamersgate email was it that you missed? Maybe I should be working with a different one.

[SysKoll]

Very interesting. Can you PM me the gmail "spam" message, including the headers? Feel free to censor the body for private data.

[tousavelo]

Hello,

I had "reply adress masking" ON.

I got two emails lost between the entry in spamgourmet and gmail, or delayed by more than 12 hours. As I haven't seen such delays since years, I report the cases now as you may still have something in your logs helping you to diagnose the issue. I don't expect you to redeliver the emails By the way, the emails were coming from a well-known company (known in Belgium that is) with pretty standard content. However, I have no way to tell if Gmail received the emails (and somehow discarded them completely in a brutal way) or if they got stuck somewhere inside spamgourmet.

The protected address is @gmail.com. The emails are nowhere to be found (not in inbox, not in spam). One email should be around 11h00am on 9/03/2013, Paris time. The other should be slightly before 23:35 on 8/03/2013, Paris time. The spamgourmet adress is proximus.XXXXXXXX@antichef.com. I could PM the full adress but I don't have copies of the lost emails so far. My diagnostic comes from the fact that I was announced those emails, I have seen that the spamgourmet counter was incremented (due to other emails on the same adress) but the emails were not in gmail.

Best regards
Olivier

PS: I have set "reply adress masking" off right before posting this.

[SysKoll]

So, another case of emails with masked sender lost when forwarded to gmail.

You did see the counter increment, right?

[tousavelo]

In reply to SysKoll » Sat Mar 09, 2013 5:48 pm

Yep, I did see the counter increment.

I received emails number 1, 2, 3, 5 and 7.

4 and 6 are missing.

Enjoy the weekend
O.

[SysKoll]

I at a a loss here. While would only some emails be discarded?

I looked up our server in various blacklists and we're currently not listed anywhere, so on what basis would email from sg be discarded?

Are you SURE the missing messages didn't end up in your gmail's spam folder?

[tousavelo]

Hello,

Thank you for your attention.

I'm 100% sure those emails were not in the Spam folder in the Gmail web interface. By the way, the emails are not available in my email client either (Thunderbird, configured in POP3). Not that I expected a difference but I checked anyway.

Each of those email is very different from the others (I was not generating reproducible test cases, with e.g. the same automated registration email being sent repeatedly). So, differences are not unexpected.

Note that in email 3 bellow that the "from" looks bad to me and deserves your attention. I think unexpected feature 1 is that there is most probably something missing before the @ "at" sign. There may be a second unexpected feature as the "display from" should probably be longer, e.g. as "Proximus – Service clientèle", instead of "Proximus - " (not sure about the accented character, just fantasizing).

Details of a series of emails, with some lost, bellow

And now, a tentative explanation. There is an old problem with the Subject line when it includes some non ASCII characters. I reported it in 2008 and Syskoll answered "I think that the code that parses the subject is a bit simplistic indeed.". See viewtopic.php?f=4&t=1125

The Subject problem still exists, as in "Information importante à propos de la securité des c (evernote: message 2 of 20) omptes Evernote - Réinitialisation des mots de passe". Note that the message count is not at the end, note that the original subject includes accented characters as ‘é’ and ‘à’. By the way, note that the spamgourmet “suffix” is almost exactly in the middle of the original subject, as if the code expected each character to take one byte and the actual encoding resulted in 2 bytes per character, or a more complex variant of that idea.

PLEASE PLEASE PLEASE READ ALSO/MOSTLY BELLOW
My bet is that the "display from" of the missing emails originally includes a non-ASCII character and that the code doesn't always fully support it. As in the problem above with the Subject. Either the code generates something weird that Gmail doesn’t accept, either the code somehow prevents those email to get out of Spamgourmet. Note that there are example with accented characters that look just fine. So, if my hint is valuable, it is only part of the story... See bellow.

Thank you for spamgourmet. After years, it's been of great use. The “Subject” problem above mostly goes unnoticed. The “From” problem is a bit more annoying though (as “reply address masking” is so handy that I wish it will be safe to reenable it soon.)

Olivier


The XXXXXX, UUUUUUU, DDDDDD, PPPPPPP etc bellow have been used to replace personal information.

Series of various emails to the same antichef.com adress. (Sorry for the use of the code style. It is not realaly adequate but it avoids the system to turn the adresses into mailtos, which are not fully displayed, at least not in Preview.)

Code: Select all

Email 1: Found in Gmail: proximus.XXXXXX.4ee6469d7c.productline#belgacom.be@ob.0sg.net
Email 2: Found in Gmail: proximus.XXXXXX.c00939ccf4.no-reply#belgacom.be@ob.0sg.net

Email 3: Found in Gmail: proximus.XXXXXX.1ba97dde26.@ob.0sg.net

Code: Select all

Email 4: Nowhere to be found in Gmail, not in Inbox, not in Spam, not in Search results
Email 5: Found in Gmail: proximus.XXXXXX.1748ccf3c9.UUUUUUUUU#DDDDDDD.be@ob.0sg.net
Email 6: Nowhere to be found in Gmail, not in Inbox, not in Spam, not in Search results
Email 7: Found in Gmail: proximus.XXXXXX.1748ccf3c9.UUUUUUUUU#DDDDDDD.be@ob.0sg.net

Extracts of email 3 after POP3 download in Thunderbird:

Received: from spamgourmet by gourmet.spamgourmet.com with local (Exim 4.69)
(envelope-from <proximus.XXXXXX.1ba97dde26.@ob.0sg.net>)
id 1UE5NG-0007D9-OM
for PPPPPPPPPPPP@gmail.com; Fri, 08 Mar 2013 22:02:46 +0000
From: "Proximus - " proximus.XXXXXX.1ba97dde26.@ob.0sg.net





For completeness, extracts of other emails, with accented character in “Display From” that look just fine, after POP3 download in Thunderbird. Note that the accented characters look OK in the "Message" window of Thunderbird but look strange in this Copy/Paste of the source of the message. (To be clear, the missing emails are missing both from Gmail and from Thunderbird.) :

Received: from spamgourmet by gourmet7.spamgourmet.com with local (Exim 4.69)
(envelope-from <ecoconso.NNNNNNN.a40b21d860.artdeco#ecoconso.be@ob.0sg.net>)
id 1TvbzB-0004vU-F5
for UUUUUUUU@gmail.com; Wed, 16 Jan 2013 23:01:33 +0000
From: "écoconso - artdeco@ecoconso.be" <ecoconso.NNNNNNN.a40b21d860.artdeco#ecoconso.be@ob.0sg.net>
Reply-To: "artdeco@ecoconso.be" ecoconso.NNNNN.a40b21d860.artdeco#ecoco ... ob.0sg.net



Received: from spamgourmet by gourmet7.spamgourmet.com with local (Exim 4.69)
(envelope-from <etoile.XXXXXX.7a2e69d2ff.quality#citroen-ipsos.com@ob.0sg.net>)
id 1Tyi6E-0003Oi-Pr
for UUUUUUUUUU@gmail.com; Fri, 25 Jan 2013 12:09:38 +0000
Reply-To: "quality@citroen-ipsos.com" <etoile.XXXXXXX.7a2e69d2ff.quality#citroen-ipsos.com@ob.0sg.net>
From: "CITROËN Satisfaction survey - quality@citroen-ipsos.com" etoile.XXXXXXX.7a2e69d2ff.quality#citro ... ob.0sg.net




Received: from spamgourmet by gourmet.spamgourmet.com with local (Exim 4.69)
(envelope-from <biblio.XXXXXX.d0b1c0c16e.bibli.arlon#bibli.cfwb.be@ob.0sg.net>)
id 1UBRg8-0008PG-Sk
for UUUUUUUUU@gmail.com; Fri, 01 Mar 2013 15:15:20 +0000
From: "Bibliothèque Communale d'Arlon - bibli.arlon@bibli.cfwb.be"
<biblio.XXXXXX.d0b1c0c16e.bibli.arlon#bibli.cfwb.be@ob.0sg.net>



UPDATE - WARNING: The text above was edited and copy/pasted with multiple applications. Including MS-Word that "improved" it... So, in some cases, the '<' and '>' ("lower than" , and "greater than" characters) around adresses have been lost in the process.

[SysKoll]

Good indo, tousavelo, thanks/

[tousavelo]

Hello,

In my wildest dreams, this problem is under active investigation. So, I tend to send my "discoveries" and thoughts as they come. Sorry for the ... spamming

So, yesterday, I showed an example with a bad From address build by spamgourmet with "Reply Adress Masking" ON, i.e.
From: "Proximus - " proximus.XXXXXX.1ba97dde26.@ob.0sg.net

I also mentioned 2 cases of lost emails, with the hypothesis that the "display From" would contain accented characters.

I now managed to get another copy of one of these lost emails (with Reply Adress Masking OFF). As opposed to my hypothesis, it does not have accented characters in the display From (details bellow). It does however have accented characters in the "Subject". It does show the spamgourmet suffix (message X of Y) much before the end of the subject string. (BTW: I think I read something about multiple lines field. The From: and Reply-To: seem to be multiple-lines.)

Cheers
Olivier

From: "Belgacom Webteam (NO REPLY)"
<mobile_complaint@skynet.be>
Reply-To: "Belgacom Webteam (NO REPLY)"
<mobile_complaint@skynet.be>
To: proximus.XXXXXX@antichef.com
Date: Mon, 11 Mar 2013 23:51:18 +0200 (GMT-2)
Subject: =?ISO-8859-1?Q?Votre_adresse_e-mail_concernant_nos_produits_et_services_?= (proximus: message 8 of 20)
=?ISO-8859-1?Q?[Probl=E8me:130311-000795]?=
MIME-Version: 1.0

By the way, here is the Subject as it gets displayed in Thunderbird. (The bold formatting is added here by myself for your convenience.):
"Votre adresse e-mail concernant nos produits et services (proximus: message 8 of 20) [Problème:130311-000795]"

[SysKoll]

Josh, we need to discuss this.

[tousavelo]

-
-
-
UPDATE 16/03/2013: I now think the text bellow the line only shows my lack of expertise and doesn't contain anything of value.

More specifically, I now think that neither the 80 nor the 64 values are worth getting nervous about. I have successfully sent an email from Outlook Web Access to Gmail, with a local-part of 200 characters.

------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi there,

I took a look at spamgourmet-2.0.3.tgz. Reply address masking seems to be related to the MASKFOWARD keyword and the getRedirectedAddress function.

getRedirectedAddress includes

my $raddr = '+' . $word . '+' . $user . '+' . $hash . '.' . $addr . '@' . $host;
if ($name && length($raddr) < 80) {
$raddr = $name . ' <' . $raddr . '>';
}
return $raddr;


Although I'm not knowledgeable about emails, nor in Perl, I have a bad feeling about the condition with 80. If longer than 80 for $raddr is a problem/risk (otherwise why would one check the length and avoid making $raddr even longer?), I don't see how the code really mitigates that risk. If $raddr is too long in the first place (before adding $name), is it really OK to proceed normally (maybe one would have to disable MASKFOWARD, although it's a tough call) ? If $raddr is "short" at first, is it really OK to add any $name regardless of the size of $name (maybe one should check the final concatenated length before proceeding with the final concatenation) ?

By the way, I took a look at http://en.wikipedia.org/wiki/Email_address. I read " The format of email addresses is local-part@domain where the local-part may be up to 64 characters long". With MASKFORWARD, the local-part gets much longer, so I guess in some cases one may hit the 64 characters "limit" (which may or may not be currently a "hard" limit, I have no clue). It was not the case with my missing email though.

The email that was missing (with MASKFOWARD ON) had (when received with MASKFORWARD OFF):
From: "Belgacom Webteam (NO REPLY)"
<mobile_complaint@skynet.be>
Reply-To: "Belgacom Webteam (NO REPLY)"
<mobile_complaint@skynet.be>

The "redirected address" may then have looked at first something like

Code: Select all

proximus.XXXXXX.a87128650b.mobile_complaint#skynet.be@ob.0sg.net

which is 64 characters long.

So, the getRedirectedAddress code would then add the $name and, as the $name is something like "Belgacom Webteam (NO REPLY)", 27 characters long (or 29 with the quotes), the final $raddr would get past the 80 boundary.

Except for the code, I personally see no reason for the 80 limit. But it was certainly included in the code for a reason.

Talking about size limits, it reminds me of another example in an earlier post where I felt something was truncated in the From (and the address itself looked corrupted):

From: "Proximus - " proximus.XXXXXX.1ba97dde26.@ob.0sg.net

Note that the two emails above are different but they come from companies with the same mindset and they may both like to use long local-parts as in "mobile-complaint".

Thank you for your interest.
Have a good day
Cheers

Olivier

[tousavelo]

-
-
-
UPDATE 16/03/2013: I now think the text bellow the line only shows my lack of expertise and doesn't contain anything of value.

More specifically, I now think that neither the 80 nor the 64 values are worth getting nervous about. I have successfully sent an email from Outlook Web Access to Gmail, with a local-part of 200 characters.

------------------------------------------------------------------------------------------------------------------------------------------------------------
Hello,

As a sidenote, for your information, I put some thought in
"By the way, I took a look at http://en.wikipedia.org/wiki/Email_address. I read " The format of email addresses is local-part@domain where the local-part may be up to 64 characters long". With MASKFORWARD, the local-part gets much longer, so I guess in some cases one may hit the 64 characters "limit" (which may or may not be currently a "hard" limit, I have no clue). "

I took a look at my history of disposable adresses and contacts to assess whether the 64 "limit" for the local-part may be reached in practice with Reply Adress Masking ON. If I combine my longest disposable address with a long contact, I can get to 70 just for the local part, so past the 64 "limit":
defensemedecinenatur.XXXXXX.1748ccf3c9.DNRC-newsletter#unitedmedia.com

The whole thing would be 81 characters:

Code: Select all

defensemedecinenatur.XXXXXX.1748ccf3c9.DNRC-newsletter#unitedmedia.com@ob.0sg.net

FYI, I have 6 disposable prefixes that are 17 or longer, with a maximum of 20 characters:
defensemedecinenatur
emusicfirststeponly
passiefhuisplatform
advertiseco2ford01
liguesdesfamilles
eurodesigncontest

But I have no clue how that "limit" is implemented in email software in general, and specifically currently for Gmail. I tryed to make some tests but I only managed to get the feeling that I somehow harmed the infrastructure of my domain host and/or that Gmail somehow listed some of my adresses as a second class citizen with longer delays for delivery.

Cheers
Olivier

EDIT:
PS: I went the last mile in checking my history and found an email of July 2011, with a Gmail protected adress that I could download in Thunderbird with the following From generated by spamgourmet:

Code: Select all

+arretsurimages+XXXXXX+a071be2fa9.gazette-noreply#arretsurimages.eu@spamgourmet.com

.
This is 83 characters long in total, with 67 for the local-part.
PS: For some reasons, I get passionate about this. If you did come up with a solution or explanation, it would help me moving on Or I could talk to my mom or shrink

[tousavelo]

Hello,

I just realize that the code extract that I commented above produces adresses like the one of 2011, with a few '+' signs as delimiters. The current adresses don't have those "+" signs though. So, that code must be outdated.

I can't remember how I got the code but it was probably from the download section.

Regards
O.

[SysKoll]

So we have several avenue of research: Size of the email address, parsing of the space continuation characters in one of the email headers.

Thanks for the detective work, Tousavelo.

[tousavelo]

Hi there,

** Regarding the hard or soft limit of 64 characters for the local-part of the address. The following says it is a "soft" limit: http://www.apps.ietf.org/rfc/rfc2821.html#sec-4.5.3. The following test got through (back and forth). From Outlook 2010, Exchange 2010 to Gmail via spamgourmet. That local part is 71 characters long (and the whole address is 83 long). It worked fine with and without accented characters in the subject. In the adress bellow, I just replaced some characters by x for privacy reasons, but the length has been preserved.

Code: Select all

From:  1---5----1----5----2.xxxxxx.f8c1c1608b.xxxxxxx.xxxxx#xxxxxxxx.xxxxxx.xx@ob.0sg.net

** I am still under the impression that the part of the spamgourmet code that generates the complex adress for address masking is in the getRedirectedAddress function in \spamgourmet\modules\Mail\Spamgourmet\Util.pm.

The supposedly current version of the code that I can find (in subversion or in spamgourmet-2.0.3) would generate addresses with "+" signs.
See line 126 in http://spamgourmet.svn.sourceforge.net/ ... iew=markup .
It goes like this
my $raddr = '+' . $word . '+' . $user . '+' . $hash . '.' . $addr . '@' . $host;

Yet, the addresses that the system currently generates don't have "+" signs. They look like:

Code: Select all

From:  "artdeco@ecoconso.be - artdeco@ecoconso.be" <ecoconso.XXXXXX.a40b21d860.artdeco#ecoconso.be@ob.0sg.net>

There may be changes deployed in production that haven't been committed in subversion.


** By the way, I will update soon the old entry about the subject line.

Thank you for your attention
Best regards
O.