Discussion re sg development. You don't have to be a developer.


Postby Guest » Fri Aug 29, 2003 1:37 pm

By: jqh1 ( Josiah Hamilton )
Re: Demonizing
2002-12-09 07:27
First, I tend to think that using the forums will be better for
discussion, simply because of the accessibility and archival
features. I'll post this message to the new sg "developers" forum as I'm
sending it. I can set up the forum to copy new messages to an address
that forwards to the three (actually four, including John Bacalle) of us.

The two major goals that Amit and I had discussed were 1) building the
spamgourmet logic into a standardized filter form so that it could be more
easily installed and distributed (via cpan, for instance) and 2) creating
a standalone daemon that could run in the place of sendmail. My original
thoughts on that were that it would still be 100% perl, running under one
of the available packages that allows for perl daemons (miniserv? is
that one?). We hadn't really contemplated a daemon version that would run
in conjunction with sendmail, but it sounds like a good idea. I've looked
previously and it seemed to me that you couldn't evoke an IPC call via
.forward, but perhaps there's another way.

What will remain after that is providing a generic user interface. The
direction this takes will depend somewhat on the "use case" aspect of the
distributed code - for instance, we could put together a UI consisting of
config files, ala procmail, or maybe something web based. Perhaps the web
based UI versions could integrate with existing packages, such as PHPNuke
or postnuke, etc. The easier it is to set up, the better.

On that note, I think the distributed versions should not require that a
dot be used for the delimiter. It'll be fairly trivial to code
this. What's keeping me from doing it on the existing service is that
there are usernames that include just about every conceivable other
delimiter character, and these would need to be removed before we allowed
and alternative delimiter there.

The spamgourmet site has been running these two years on a basic
webhosting account that I have which costs about $10/month. I don't have
root on the box. (I've thoroughly revieed security on the box, and found
it to be very good). I'd like to make setting something like that up
easier to encourage others to do the same. Folks who have control over
their own servers could opt for the standalone version if they liked. So
basically, I'd like the code to remain accessible to non-root users, but
provide additional functionality (or performance) options for root users.


> Hi Josh,
> In order to get my feet wet, I made some small performance improvements in a
> couple of routines in
> According to my discussion with Amit, the current major project is to
> demonize Did you already decide on the protocol you're going to
> use, i.e., how spameater will receive messages from the C prog launched by
> .forward? Anything I should know?
> Also, do you want these technical decisions to be held through email or
> through forums?
> Thanks,

By: syskoll ( Fred )
Re: Demonizing
2002-12-09 18:27

Sounds like a bold plan. I like the idea of helping people clone SG.

I am a tad overwhelmed by the magnitude of the task. So here are a few dumb questions.

You write, "1) building the spamgourmet logic into a standardized filter form". What form? Do you mean a command-line filter suitable to pipe, or something else?

"2) creating a standalone daemon that could run in the place of sendmail.". Do you want to recreate all the functionalities of sendmail?

What I had in mind was a daemonized version that would run in conjunction with sendmail and accept input from .forward. The demonized would wait for incoming messages to process. The .forward would contain
"| /full/path/to/Forwarder"

where Forwarder is a small and fast program that would send the message to though IPC (probably sockets).

Thanks in advance for the explanations,

-- SysKoll

By: maratheamit ( Amit Marathe )
Re: Demonizing
2002-12-09 19:12
I also had in mind a daemon which would work in conjunction with sendmail through a .forward file and a small go-between program. I am not convinced that we need to write a sendmail replacement.

As for your first question, the filter Josh is talking about is Mail::Audit. It is a perl package which simplifies the construction of mail filters (which spamgourmet is). If you noticed, I have already made the changes in CVS. Now, it's a matter of testing them before deployment.

-- Amit

By: syskoll ( Fred )
Forwarder program
2002-12-09 21:19

Sounds good. About this go-between program (which I call Forwarder), how about this basic spec:

- C code for raw speed
- invoked by "| /path/of/program" in .forward, hence gets the email message in stdin
- Checks using some IPC to see if is alive and listening (IPC faster than reading some file)
- If not, starts and wait for it to initialiaze
- Sends the email message to through a FIFO (faster than IP socket) or maybe a msgqueue.

Also, for testability, I'd want Forwarder to allow command-line parms to turn on various debugging traces such,as in
"| /path/of/program -t 1 -l /tmp/logfile" in .forward to turn on traces level 1 and write them in /tmp/logfile.

Your thought?

-- SysKoll

By: maratheamit ( Amit Marathe )
Re: Demonizing
2002-12-11 07:29
I like the idea of the Forwarder handling command line options. However, for portability reasons I think we should stick to plain sockets rather than IPC. The performance gain is not enough (in my opinion) to offset putting in code which may need tweaking to compile on other systems.

On the same note, we should provide the Forwarder as both a C and perl program. That's because SG should be easy to install and some people may not have access to a shell account necessary to run a compiler. The makefile associated with the C program would also need to be changed for another system and some users may not have the expertise to do that.

So my suggestion is that we provide the Forwarder as a perl script. If people are still worried about efficiency we can point them to our C version of the script which they would be responsible for compiling on their system.

I realize that a perl Forwarder would be less efficient than a C one but then just daemonizing will result in a big jump in efficiency. And once we have SG running as a daemon there are lots of tricks we can play to make it run even faster (like caching tuples to avoid going to the database every time).

-- Amit

Return to Developers

Who is online

Users browsing this forum: No registered users and 1 guest