New Server

Discussion re sg development. You don't have to be a developer.

New Server

Postby Guest » Fri Aug 29, 2003 1:47 pm

By: maratheamit ( Amit Marathe )
New Server
2003-07-31 08:43
Josh, what is on your to-do list now that mail processing has been moved to the new server? I think that we should now move to daemonize the spameater process for efficiency, using Syskoll's code.

-- Amit

By: jqh1 ( Josiah Hamilton )
RE: New Server
2003-08-17 05:36
Once again, sorry for my scarcity - work has gotten crazier, more layoffs, more craziness, followed by layoffs and additional craziness. Lots of that going around, I know...

I'm still setting up the new server -- I think it has a memory problem, since it's been going down some. I ordered a memory upgrade that will involve replacing all the existing memory modules, so that should help.

I've more or less been in a holding pattern waiting for that to happen. As soon as we're stable, then definitely let's install syskoll's daemon code.

And again, I'll get off my *ss and package some more releases.

I had an idea for message throttling -- I had to do a hack implementation of it briefly before the server move - when the server has come under attack, it's usually been with just a few sg user accounts involved. If we implement syskoll's proposed throttling method in the database, we can then have a fallback to a file once the threshold is hit -- this way, the script won't even make a database connection for the accounts that are under attack (until the attack stops and the file time expires). I hard coded this for one account during the DOS attack in early July, and it brought the server back up.

I was also contacted by someone who's interested in putting together a PHP Nuke module for an sg user interface -- if we could get an easy install going on with Nuke and/or the other widely distributed community/content packages, I'll bet we'd finally start seeing some independent servers (there are a couple now, but I think they're more or less private).

Are you guys getting email for these posts? Seems like I'm not anymore...

By: maratheamit ( Amit Marathe )
RE: New Server
2003-08-18 05:41
I too wasn't being notified of new postings. I had to click on "Monitor this forum" to re-start those emails.

-- Amit

By: syskoll ( Fred )
RE: New Server
2003-08-18 12:41
I had to click on "Monitor this forum" again, too.

PHP is a good way of getting an easy-to-maintain user interface, I second that.

-- SysKoll

By: syskoll ( Fred )
Test server?
2003-08-18 13:02

I suggest to try the daemonized code on a test server. Then we can try to implement the throttling. Or do you prefer to start working on the throttling first?

By: jqh1 ( Josiah Hamilton )
RE: New Server
2003-08-18 16:39
Yeah -- some sort of formalized load testing would be good. I plan on giving up the current test server soon, but maybe we can try it on there first... Can you guys still log on there?

Also, I've turned back on the eaten message log on a "feature" basis (and there's currently no way to turn on the feature through the GUI) -- that is, spameater only does the extra work for the log if you have the feature enabled. I took a guess at which spamgourmet accounts were yours (syskoll and amit) and enabled the feature there. I'm starting to bench it again, and I may put a "use bench;" version of the script briefly into production. My feeling is that it's a noticeable premium, and the difference is human-perceivable (at least I think), but we'll see. The other consideration is that we might be able to handle the extra load now with the new server, especially if we have the breaker-box style throttling using a file.

BTW, I'm waiting for HE support to schedule the memory upgrade - it should be finished soon.

By: syskoll ( Fred )
eaten log feature looks great
2003-08-18 16:58
The eaten message feature seems to work great on my SysKoll account. I verified that I could still logon to the SG shell account.

By: maratheamit ( Amit Marathe )
RE: New Server
2003-08-18 19:47
Storing the eaten message logs in the database is always going to incur a performance hit. We should move to storing these logs in the filesystem (refer to our discussion in this forum towards the end of last year). I assume that space is not too tight with the new server...

-- Amit

By: jqh1 ( Josiah Hamilton )
RE: New Server
2003-08-19 16:40
yeargghh. Load average on the machine peaked hard today, and sendmail started refusing connections, leading to delayed delivery (it's catching up now). I assumed it was the eaten message log code (even though it's only on for us three) but I don't think that was it. I did roll back, for now, though.

What was weird is that the load average was pegged at 47.0% (!!! - usually runs at ~0.02%) even though nothing was happening on the server, and top only showed a few fractions of a percent used in user and system. Sendmail was running, although it was refusing connections, as was mysql, etc.

I rebooted the box and it's back in action -- load average started out high (~20%) and is converging back to zero as the box catches up on queued mail and deferred connections from upstream servers.

I looked in the syslog and saw this:

Aug 19 09:03:50 gourmet kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000034
Aug 19 09:03:50 gourmet kernel: printing eip:
Aug 19 09:03:50 gourmet kernel: c0149401
Aug 19 09:03:50 gourmet kernel: *pde = 00000000
Aug 19 09:03:50 gourmet kernel: Oops: 0000
Aug 19 09:03:50 gourmet kernel: CPU: 0
Aug 19 09:03:50 gourmet kernel: EIP: 0010:[<c0149401>] Not tainted
Aug 19 09:03:50 gourmet kernel: EFLAGS: 00010217
Aug 19 09:03:50 gourmet kernel: eax: dff73de8 ebx: 00000000 ecx: 00000010 edx: dff00000
Aug 19 09:03:50 gourmet kernel: esi: fffffff0 edi: df334350 ebp: e240e354 esp: d0627f30
Aug 19 09:03:50 gourmet kernel: ds: 0018 es: 0018 ss: 0018
Aug 19 09:03:50 gourmet kernel: Process sendmail (pid: 19489, stackpage=d0627000)
Aug 19 09:03:50 gourmet kernel: Stack: c0140290 fffffff0 dff73de8 c54ec003 00000010 d0627f98 df2c6040 df334350
Aug 19 09:03:50 gourmet kernel: d0627f98 c0140290 df334350 d0627f98 00000000 c0140e47 df334350 d0627f98
Aug 19 09:03:50 gourmet kernel: 00000000 c54ec000 ffffffeb d0627f90 bfffb998 c014213b d0627f98 df334350
Aug 19 09:03:50 gourmet kernel: Call Trace: [<c0140290>] [<c0140290>] [<c0140e47>] [<c014213b>] [<c0108f13>]
Aug 19 09:03:50 gourmet kernel:
Aug 19 09:03:50 gourmet kernel: Code: 39 6e 44 8b 1b 75 e8 8b 7c 24 28 39 7e 0c 75 df 8b 47 4c 85

(this was the last entry - there's a lot of it). I think maybe this is related to the memory problem I mentioned -- still no upgrade. Maybe it was bad RAM that was responsible for returning the 47% figure for CPU load even though all was quiet??

I'm also looking for indicators of a DOS attack, but haven't been able to match this up with anything.

Anyway, crappy day...

By: syskoll ( Fred )
RE: New Server
2003-08-19 21:51
Yikes! Looks like the kernel had a fit on a sendmail system call. I really hope that it's a RAM problem -- much easier and faster to fix than a kernel bug.

By: nobody ( Nobody/Anonymous )
RE: New Server
2003-08-20 04:07
Yeah, the other cause I've read about is that the kernel doesn't "match" the machine that it's on -- these commentators suggest that you compile the kernel on the machine. HE did the set up for this machine, so I'm not sure how they did it. If the problem persists after the upgrade, I suppose I can always get the kernel sources (no SCO code, of course) and build locally, like we used to have to do always.

By: jqh1 ( Josiah Hamilton )
RE: New Server
2003-08-20 14:03
back up with the new memory - So far nothing negative in syslog. We'll see how it goes...

Return to Developers

Who is online

Users browsing this forum: No registered users and 2 guests