more about pressure valve throttling

Discussion re sg development. You don't have to be a developer.

more about pressure valve throttling

Postby josh » Sun Oct 26, 2003 7:17 am

I just got throttling implemented at the db level - that was the hard part. I'll do the file part soon. Then I'll merge :? and check in.

The database throttle will basically reduce the overhead to one SELECT statement (and no DML statements), but we still have to create the database connection, which I think is heavier than the sum of all the statements -- btw, should we use a connection pool when the thing is daemonized?

I have the send throttle set to 50/hour and the receive throttle set to 100/hour.

Does that sound reasonable? I'd like to set the thresholds so high that we don't need to mention it to Joe Average User (simply put it in the FAQ), but not so high that they don't provide much benefit.

If there's a steady stream of messages pointed at the server, say 3 per second for a particular user, I guess this would work: at the top of the hour, we'd see about 33.3 seconds of hard work, followed by almost an hour of quiet, then another 33.3 seconds of hard work, etc.

Any comments on this? Using the receive throttle may mean that you could trash an account by overusing it -- that is, you've done nothing out of the ordinary, but the account is getting hit so hard (with messages that would be eaten) that you can't get a good message through. 100/hour is a lot, though... (now, about 45 minutes into the game, several accounts are in the 50-60 range)
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm

Postby SysKoll » Sun Oct 26, 2003 3:48 pm

Josh,

Great job!

A send throttle of 50/hr is reasonable, so double to 100/hr it to make sure it matches even unreasonale requests.

As for receive, I think it should be increased. If you're seeing accounts that are in the 60 messages per 45 mins range, that's 90 messages/hr already. Considering that spam is not decreasing, we'll see maxed-out accounts with lost legit messages very soon. So SG should increase its receive limit to about 400/hr.

Don't forget to log a message when a specific account is maxed out. Initially, we'd need to investiate. Oh, and please post the code for review!
-- SysKoll
SysKoll
 
Posts: 893
Joined: Thu Aug 28, 2003 9:24 pm

Postby josh » Sun Oct 26, 2003 5:25 pm

sure -- I added the ThrottleCounts and Times to the columns that get selected when we pull out User info -- I won't post those mundane changes. The code grabs send throttle columns when sending and rec throttle columns when receiving. I also added 4 attributes to the config file - $maxsendcount, $sendthrottleinterval, $maxreccount, $recthrottleinterval (intervals in seconds).

$now is set to time() near the top of the code.

Then, as an additional threshold conditon for sending and receiving, it has this (for sending, eg):

Code: Select all
&& ($SendThrottleCount < $maxsendcount || ($SendThrottleTime < $now - $sendthrottleinterval))


If the condition is not met, we go to "skip" mode, which prevents any further database activity from taking place.

To maintain the count and time for sending, I just added an update statement that follows the mail send.

To maintain the count and time for receiving, I added $throttleTime and $throttleCount as args to the setCount(...) sub:

Code: Select all
sub setCount { 
  my $db = shift;  # database handle
  my $UserID = shift;  # user at issue, 0 if none
  my $EmailID = shift; # Disposable address at issue, 0 if none
  my $messagetype = shift;  # 0 if spam, non0 if not spam, converted to field name to update
  my $throttleTime = shift;
  my $throttleCount = shift;

    # if it was spam, we'll upate the "eaten message log"
  my $from = shift;
  my $for = shift;
  my $updatelog = shift; # whether we should update the EML or not


Then, this check runs:

Code: Select all
    if ($throttleTime < $now - $recthrottleinterval) {
      $throttleTime = $now;
      $throttleCount = 1;
    } else {
      $throttleCount++;
    }


and the two columns are simply updated with those values along with the same update statement that maintains the user stats. ($now is actually redetermined in the subroutine scope, since it's not globally scoped - I can live with that, though)

Neither the time nor count for sending or receiving is updated if we changed over to "skip" mode, of course.


I'll start working on the file part soon. I suppose we could use a multiplier of the intervals that could be set in the config file, too: for instance, if the multiple were three, and the sendInterval were 1hour, then exceeding the send count would result in a 3 hour file-based lockout.

We're now dangerously close to adding the most requested feature -- sending the first message. I'll start another thread for that.
josh
 
Posts: 1371
Joined: Fri Aug 29, 2003 2:28 pm


Return to Developers

Who is online

Users browsing this forum: No registered users and 13 guests

cron