hitting the ceiling at he.net...

Discussion re sg development. You don't have to be a developer.

hitting the ceiling at he.net...

Postby Guest » Fri Aug 29, 2003 1:47 pm

By: jqh1 ( Josiah Hamilton )
hitting the ceiling at he.net...
2003-03-25 16:20
Recently, the sg production server has been experiencing some errors caused by mysql rejecting connections ("too many connections") -- it was bound to happen sooner or later.

I'm shopping around for a new provider -- colo or dedicated at he.net would cost several hundred dollars a month, which is too much for me :)

I'm looking at:
-- it looks like they offer a dedicated linux instance with 4 gigs of disk space (probably a virtualized server?) for $45/month to open source developers, and the package looks pretty good in other respects, too.

Do you all have any opinions on this?


By: jqh1 ( Josiah Hamilton )
2003-03-25 16:58
would this work as a workaround?

Currently, the connection code is:

$db = DBI->connect($dsn,$dbuser,$dbpassword) or die "$DBI::errstr\n";

if, instead, it was:

my connectTries = 0;
while (!$db && connectTries < 10) {
$db = DBI->connect($dsn,$dbuser,$dbpassword);
if (!$db) {
sleep 6;
connectTries ++;
if (!$db && connectTries >= 10) {
die "could not connect to database after a minute of trying: $DBI::errstr\n";

Any problems there?

By: maratheamit ( Amit Marathe )
RE: hitting the ceiling at he.net...
2003-03-27 14:24
i don't see how the workaround would help. the problem seems to be that multiple spameater.pl processes are trying to open a connection to the database. correct me if i am wrong, but i am assuming that sendmail forks off a seperate process to handle each message coming into spamgourmet.com and that these messages can arrive in parallel.

if that's the case, making a process wait for a fixed amount of time is only going to delay the contention. changing the wait time to be a random number would be slightly better but still a stop-gap
measure (and it won't help at all if the rate of message arrival is uniform rather than bursty).

a long term solution would be to reduce the number of processes launched in parallel. the daemonize feature would be ideal right now. another fix i can suggest is to change .forward to write each message to a different file and run spameater in batch mode every 10-15 minutes on all the accumulated files.

-- amit

By: maratheamit ( Amit Marathe )
RE: hitting the ceiling at he.net...
2003-03-30 12:45
the package at johncompanies looks good. but given that it's involves a steep increase over the current hosting fees it probably means you have to ask for donations from SG users.

that may not be a bad idea in the long run as even a token donation from a significant fraction of users will enable us to get better hosting resources. my only concern here is that users not be required to pay for the service initially (at least a few months) as that would be a major impediment in SG's acceptance.

-- amit

By: maratheamit ( Amit Marathe )
RE: hitting the ceiling at he.net...
2003-03-30 12:49
given that the message is silently dropped when mysql refuses a connection, i would say that a temporary fix is urgently needed.

josh, let me know if you would like me to code up the solution i described above.

-- amit

By: syskoll ( Fred )
RE: hitting the ceiling at he.net...
2003-07-03 14:22
Amit, the demonizer code has been available for a while, written by yours truly. Josh, will you have time to take a look at it and maybe put it on a test server soon?

-- SysKoll

Return to Developers

Who is online

Users browsing this forum: No registered users and 3 guests