Re: Stuff running slooow

> -----Original Message-----
> From: Jim C. Nasby [mailto:jnasby(at)pervasive(dot)com]
> Sent: 25 August 2005 21:24
> To: Dave Page
> Cc: mha(at)sollentuna(dot)net; scrappy(at)postgresql(dot)org;
> pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] Stuff running slooow
>
> So I take it the bottleneck is the box running the mailing list?

Usually that, or av.hub.org which does the centralised anti virus/anti
spam (iirc).

Regards, Dave.

On Thu, Aug 25, 2005 at 09:26:25PM +0100, Dave Page wrote:
> > So I take it the bottleneck is the box running the mailing list?
>
> Usually that, or av.hub.org which does the centralised anti virus/anti
> spam (iirc).

Does it scan every single incomming email? It might make more sense to
have the mailing list software first validate that the email is from a
valid subscriber.

What exactly runs on svr1, since that's where the bottleneck was
yesterday?
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com 512-569-9461

On Thu, 25 Aug 2005, Jim C. Nasby wrote:

> On Thu, Aug 25, 2005 at 09:26:25PM +0100, Dave Page wrote:
>>> So I take it the bottleneck is the box running the mailing list?
>>
>> Usually that, or av.hub.org which does the centralised anti virus/anti
>> spam (iirc).
>
> Does it scan every single incomming email? It might make more sense to
> have the mailing list software first validate that the email is from a
> valid subscriber.

I have just about every anti- filter that I can find enabled on postfix
itself, but its postfix itself that does the content filtering, not the
mailing list software ... so thet anti-virus is hit before the mailing
list software even sees it ... we also have spamassassin running, and
filters in place to reject any messages that trigger that ...

As a couple of ppl have found out by becoming 'moderators' for the mailing
lists, there are *alot* of messages through the server that aren't list
subscribers, but are legit emails ...

The process for email right now is:

scan for virus -> analyze/tag for spam -> pass to majordomo

majordomo then checks the spam tags and disregards based on a set of
rules, and what is left either has to wait for moderator approval/reject
*or* send onto the list, depending on if someone is subscribed or not ...

If y'all would like, I can eliminate the anti-virus/anti-spam checks and
just let it all go through though ... *evil grin*

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Marc G. Fournier wrote:

> As a couple of ppl have found out by becoming 'moderators' for the
> mailing lists, there are *alot* of messages through the server that
> aren't list subscribers, but are legit emails ...

Perhaps that shouldn't be allowed? Would it help things if all
non-subscriber emails are just bounced / dropped immediately, before
anti-virus etc... Seems this would save a lot of CPU time and more
importantly people time reviewing potentially legit emails.

> If y'all would like, I can eliminate the anti-virus/anti-spam checks and
> just let it all go through though ... *evil grin*

Would not bother me in the least. I have protective measures as I am
sure most others do as well. :)

Sincerely,

Joshua D. Drake

>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org

--
Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240
PostgreSQL Replication, Consulting, Custom Programming, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/

On Thu, 25 Aug 2005, Matthew T. O'Connor wrote:

> Marc G. Fournier wrote:
>
>> As a couple of ppl have found out by becoming 'moderators' for the mailing
>> lists, there are *alot* of messages through the server that aren't list
>> subscribers, but are legit emails ...
>
>
> Perhaps that shouldn't be allowed? Would it help things if all
> non-subscriber emails are just bounced / dropped immediately, before
> anti-virus etc... Seems this would save a lot of CPU time and more
> importantly people time reviewing potentially legit emails.

Not sure how you would accomplish this, since postfix is what handles the
anti-virus/anti-spam processing, before the list software even sees it ...
and, since not all mail through that system is for the lists, you can't
just drop all messages not from subscribers ...

the issues, though, aren't with the anti-virus/anti-spam ... somehow, the
messages seem to be getting 'stuck' within the list software itself, which
is what I'm trying to trace through ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

On Thu, 25 Aug 2005, Joshua D. Drake wrote:

>
>> If y'all would like, I can eliminate the anti-virus/anti-spam checks and
>> just let it all go through though ... *evil grin*
>
> Would not bother me in the least. I have protective measures as I am sure
> most others do as well. :)

Remembering back to the time I *oopsed* and approved all messages in the
moderator queue, and the # of ppl emailing me about getting a whack of
spam, I don't imagine everyone has such in place :)

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Marc G. Fournier wrote:
> On Thu, 25 Aug 2005, Joshua D. Drake wrote:
>
>>
>>> If y'all would like, I can eliminate the anti-virus/anti-spam checks
>>> and just let it all go through though ... *evil grin*
>>
>>
>> Would not bother me in the least. I have protective measures as I am
>> sure most others do as well. :)
>
>
> Remembering back to the time I *oopsed* and approved all messages in the
> moderator queue, and the # of ppl emailing me about getting a whack of
> spam, I don't imagine everyone has such in place :)

O.k. that is probably true, but Matt had a good suggestion. If you are
not subscribed it immediately bounces. I think that is a very good idea.
It would take some load off of the system and the moderaters.

Sincerely,

Joshua D. Drake

>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

--
Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240
PostgreSQL Replication, Consulting, Custom Programming, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/

On Thu, Aug 25, 2005 at 15:01:25 -0700,
"Joshua D. Drake" <jd(at)commandprompt(dot)com> wrote:
>
> O.k. that is probably true, but Matt had a good suggestion. If you are
> not subscribed it immediately bounces. I think that is a very good idea.
> It would take some load off of the system and the moderaters.

That makes the lists less usable for people asking questions.

Are messages from usenet still being gated to the lists? If so that will
also be affected by such a change.

"Dave Page" <dpage(at)vale-housing(dot)co(dot)uk> writes:
>> So I take it the bottleneck is the box running the mailing list?

> Usually that, or av.hub.org which does the centralised anti virus/anti
> spam (iirc).

Yesterday's problem seemed to be av.hub.org; svr1 was pretty nearly idle
as far as I could tell. I don't have a login on av to see what
conditions were like there, though.

regards, tom lane

On Thu, 25 Aug 2005, Bruno Wolff III wrote:

> On Thu, Aug 25, 2005 at 15:01:25 -0700,
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> wrote:
>>
>> O.k. that is probably true, but Matt had a good suggestion. If you are
>> not subscribed it immediately bounces. I think that is a very good idea.
>> It would take some load off of the system and the moderaters.
>
> That makes the lists less usable for people asking questions.
>
> Are messages from usenet still being gated to the lists? If so that will
> also be affected by such a change.

Two reasons why 'auto-bouncing' won't work ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

On Thu, Aug 25, 2005 at 02:45:02PM -0700, Joshua D. Drake wrote:
>
> >If y'all would like, I can eliminate the anti-virus/anti-spam checks and
> >just let it all go through though ... *evil grin*
>
> Would not bother me in the least. I have protective measures as I am
> sure most others do as well. :)

The archives would fill with junk. From skimming other projects'
archives, IMHO that is very undesirable.

--
Alvaro Herrera (<alvherre[a]alvh.no-ip.org>)
"The important things in the world are problems with society that we don't
understand at all. The machines will become more complicated but they won't
be more complicated than the societies that run them." (Freeman Dyson)

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> O.k. that is probably true, but Matt had a good suggestion. If you are
> not subscribed it immediately bounces. I think that is a very good idea.
> It would take some load off of the system and the moderaters.

That won't do, as some other folks noted. But what I'd really like to
see is a hack that, when someone subscribes to a list, goes through the
moderator queue and auto-approves any pending messages from that
someone.

What we see way way too much of is the following sequence of events:

1. Newbie sends a question to a list.

2. Question goes into the moderator's queue because it's from a non
subscriber.

3. Newbie figures this out (maybe right away or maybe not), subscribes,
and reposts his question.

4. People answer.

5. Some while later (usually several days, which means that Marc is
badly overworked :-(), the original question gets approved and
we see a duplicate appearing on the list.

There is nothing optimal about this from the point of view of the
newbie, nor the moderator, nor the list membership (who have to be
able to recognize delayed duplicate questions when they see 'em).
Plus it clutters the archives.

An auto-approval mechanism would fix all this (though we'd probably
need to add something to the standard list-welcome message mentioning
that you shouldn't repeat any questions you already sent in). I have
no idea how hard it is to do, but it sure seems like it would make
things more pleasant all around.

regards, tom lane

I've forwarded this onto the Mj2 Developers ... it might even be doable
now, they've built a, at times, painfully configurable system ...

On Fri, 26 Aug 2005, Tom Lane wrote:

> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>> O.k. that is probably true, but Matt had a good suggestion. If you are
>> not subscribed it immediately bounces. I think that is a very good idea.
>> It would take some load off of the system and the moderaters.
>
> That won't do, as some other folks noted. But what I'd really like to
> see is a hack that, when someone subscribes to a list, goes through the
> moderator queue and auto-approves any pending messages from that
> someone.
>
> What we see way way too much of is the following sequence of events:
>
> 1. Newbie sends a question to a list.
>
> 2. Question goes into the moderator's queue because it's from a non
> subscriber.
>
> 3. Newbie figures this out (maybe right away or maybe not), subscribes,
> and reposts his question.
>
> 4. People answer.
>
> 5. Some while later (usually several days, which means that Marc is
> badly overworked :-(), the original question gets approved and
> we see a duplicate appearing on the list.
>
> There is nothing optimal about this from the point of view of the
> newbie, nor the moderator, nor the list membership (who have to be
> able to recognize delayed duplicate questions when they see 'em).
> Plus it clutters the archives.
>
> An auto-approval mechanism would fix all this (though we'd probably
> need to add something to the standard list-welcome message mentioning
> that you shouldn't repeat any questions you already sent in). I have
> no idea how hard it is to do, but it sure seems like it would make
> things more pleasant all around.
>
> regards, tom lane
>

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Tom Lane wrote:

> That won't do, as some other folks noted. But what I'd really like to
> see is a hack that, when someone subscribes to a list, goes through the
> moderator queue and auto-approves any pending messages from that
> someone.

If it's possible, cool. What I have seen from other mailing lists (possibly
newer versions of the same software?) is a "Cancel message" link in the
e-mail telling about moderation status.

So one can cancel the message, subscribe to the list and send the message
again.

At least better than what we have now.

Best Regards,
Michael Paesold

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> 5. Some while later (usually several days, which means that Marc is
> badly overworked :-(), the original question gets approved and
> we see a duplicate appearing on the list.

The several days should be a thing of the past now. Most queues get
cleaned out at least once a day. The duplicate thing is a pain, but
I'd rather err on the side of having two messages than missing one
entirely.

> An auto-approval mechanism would fix all this (though we'd probably
> need to add something to the standard list-welcome message mentioning
> that you shouldn't repeat any questions you already sent in). I have
> no idea how hard it is to do, but it sure seems like it would make
> things more pleasant all around.

Even just adding that text to the welcome message would help a lot
for the current system.

- --
Greg Sabino Mullane greg(at)turnstep(dot)com
PGP Key: 0x14964AC8 200508261048
https://www.biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAkMPLI4ACgkQvJuQZxSWSsiqnwCeO+xpU///TTXxCqXt1MRWu2Im
TAkAoK6OliABug6B2Tej0ktSVA2JpDlA
=e0rZ
-----END PGP SIGNATURE-----

alvherre(at)alvh(dot)no-ip(dot)org (Alvaro Herrera) writes:
> On Thu, Aug 25, 2005 at 02:45:02PM -0700, Joshua D. Drake wrote:
>>
>> >If y'all would like, I can eliminate the anti-virus/anti-spam checks and
>> >just let it all go through though ... *evil grin*
>>
>> Would not bother me in the least. I have protective measures as I am
>> sure most others do as well. :)
>
> The archives would fill with junk. From skimming other projects'
> archives, IMHO that is very undesirable.

We'd discover ourselves inundated with questions about Dueling Banjos...
--
let name="cbbrowne" and tld="acm.org" in name ^ "@" ^ tld;;
http://cbbrowne.com/info/spiritual.html
"Parentheses? What parentheses? I haven't noticed any parentheses
since my first month of Lisp programming. I like to ask people who
complain about parentheses in Lisp if they are bothered by all the
spaces between words in a newspaper..."
-- Kenny Tilton <tilt(at)liii(dot)com>

On Fri, Aug 26, 2005 at 11:42:04AM -0400, Chris Browne wrote:
> alvherre(at)alvh(dot)no-ip(dot)org (Alvaro Herrera) writes:
> > On Thu, Aug 25, 2005 at 02:45:02PM -0700, Joshua D. Drake wrote:
> >>
> >> >If y'all would like, I can eliminate the anti-virus/anti-spam
> >> >checks and just let it all go through though ... *evil grin*
> >>
> >> Would not bother me in the least. I have protective measures as I
> >> am sure most others do as well. :)
> >
> > The archives would fill with junk. From skimming other projects'
> > archives, IMHO that is very undesirable.
>
> We'd discover ourselves inundated with questions about Dueling
> Banjos...

This should explain it:

http://homepages.nildram.co.uk/~blagger/the_duel.html

Cheers,
D
--
David Fetter david(at)fetter(dot)org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!