Re: Maia Mailgard down?

Marc,

josh(at)postgresql(dot)org is getting a spam every 3-6 minutes. Is our spam
filter down?

--Josh

On Fri, 28 Jul 2006, Josh Berkus wrote:

> Marc,
>
> josh(at)postgresql(dot)org is getting a spam every 3-6 minutes. Is our spam filter
> down?

Email goes down when maia goes down, so it is definitely up and running
... but, you need to retrain yourself, as due to a problem with the
database, we've had to reload the system, so you need to relogin, set your
settings and re-train the bayes component ... apologies, I thought I had
sent out a notice to this list about it, but in the rush, appear to have
forgotten :(

note for *anyone* that has a mailbox @postgresql.org, you will need to
login to http://webmail.postgresql.org/maia to set this up, including all
of the various regional press contacts ... Josh, not sure where I would
send to get ahold of all of these ...

Also, once question ... does *anyone* know who david(at)postgresql(dot)org is???
If nobody does, I'll delete it, since its got a major full mailbox, and
doesn't look like any of it has ever been read ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

Marc,

> note for *anyone* that has a mailbox @postgresql.org, you will need to
> login to http://webmail.postgresql.org/maia to set this up, including
> all of the various regional press contacts ... Josh, not sure where I
> would send to get ahold of all of these ...

Isn't there some what (via Horde) that you can sent out an e-mail to all
account-holders on @postgresql.org?

And no, I don't know who david@ belongs to; feel free to delete it.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Folks,

Is there any way we can return to whatever we were using before Maia mailgard?
Maia is patently NOT working ... it's fundamentally broken in some way, and
isn't blocking any spam. The interface for Baysian filtering is also awkward
and darned near unusable. In a couple months we're going to be trying to do
the PostgreSQL 8.2 release which means that we need the @postgresql.org
e-mail addresses working ... and right now they're all receiving a flood of
unfiltered spam, making them unusable.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

On Sat, 5 Aug 2006, Josh Berkus wrote:

> Folks,
>
> Is there any way we can return to whatever we were using before Maia
> mailgard? Maia is patently NOT working ... it's fundamentally broken in
> some way, and isn't blocking any spam. The interface for Baysian
> filtering is also awkward and darned near unusable. In a couple months
> we're going to be trying to do the PostgreSQL 8.2 release which means
> that we need the @postgresql.org e-mail addresses working ... and right
> now they're all receiving a flood of unfiltered spam, making them
> unusable.

Actually, all MAIA is is a front-end over what we were using before
(amavisd + spamassassin) to allow individual mailboxes to configure
themselves, instead of one big site-wide setting ... I'm getting a
*trickle* of spam right now, but I've spent the past week or so going
through and training the Bayesian database ... in fact, under the old
system, those 46 messages that are currently quarantined in your cache
would have been delivered, instead of quarantined ... in the
@postgresql.org one that I monitor, there are >7k messages currently in
spam-quarantine that normally would have been passed through to the system
...

Not sure what you are finding awkward about the interface, but if you go
in and just turn on all the various checks, but *disable* the
quarantining, so that it just label's and passing everything over to you
to filter, you will have exactly the same thing in place that was there
before we added the interface for per-user settings ...

Please note ... at *no time* did we ever reject spam for any mailbox,
except those for the mailing lists themselves, which majordomo is setup to
do ... so if you are seeing more now then before, its because we are being
hit with more now that are passing through spamassassin ...

Note that unlike the ~60 messages in the cache for your mailbox right now
(12 unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go
through is right now sitting at ~10k, and I generally spend about a half
hour in the evening going through it ... I generally care about
unconfirmed-spam between 5 and 10, ignoring most above that ... and
unconfirmed non-spam between 0 and 5 ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

Just to give some followup numbers, between both servers that are
currently doing Spam/Virus/MAIA, here is what it has seen for josh(at)post
since midnight:

# grep josh(at)post /var/log/amavis | awk '{print $8}' | sort | uniq -c
2 CLEAN,
4 INFECTED
20 SPAM,

# grep josh(at)post /var/log/amavis | awk '{print $8}' | sort | uniq -c
4 CLEAN,
4 INFECTED
4 SPAM,

So, not 100% certain where josh is seeing this "flood" ...

... but, just in case, I've raised the timeout on both the razor2 and
pyzor checks from 10 sec to 60 sec ... see if that helps any ...

On Sun, 6 Aug 2006, Marc G. Fournier wrote:

> On Sat, 5 Aug 2006, Josh Berkus wrote:
>
>> Folks,
>>
>> Is there any way we can return to whatever we were using before Maia
>> mailgard? Maia is patently NOT working ... it's fundamentally broken in
>> some way, and isn't blocking any spam. The interface for Baysian filtering
>> is also awkward and darned near unusable. In a couple months we're going
>> to be trying to do the PostgreSQL 8.2 release which means that we need the
>> @postgresql.org e-mail addresses working ... and right now they're all
>> receiving a flood of unfiltered spam, making them unusable.
>
> Actually, all MAIA is is a front-end over what we were using before (amavisd
> + spamassassin) to allow individual mailboxes to configure themselves,
> instead of one big site-wide setting ... I'm getting a *trickle* of spam
> right now, but I've spent the past week or so going through and training the
> Bayesian database ... in fact, under the old system, those 46 messages that
> are currently quarantined in your cache would have been delivered, instead of
> quarantined ... in the @postgresql.org one that I monitor, there are >7k
> messages currently in spam-quarantine that normally would have been passed
> through to the system ...
>
> Not sure what you are finding awkward about the interface, but if you go in
> and just turn on all the various checks, but *disable* the quarantining, so
> that it just label's and passing everything over to you to filter, you will
> have exactly the same thing in place that was there before we added the
> interface for per-user settings ...
>
> Please note ... at *no time* did we ever reject spam for any mailbox, except
> those for the mailing lists themselves, which majordomo is setup to do ... so
> if you are seeing more now then before, its because we are being hit with
> more now that are passing through spamassassin ...
>
> Note that unlike the ~60 messages in the cache for your mailbox right now (12
> unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go through is
> right now sitting at ~10k, and I generally spend about a half hour in the
> evening going through it ... I generally care about unconfirmed-spam between
> 5 and 10, ignoring most above that ... and unconfirmed non-spam between 0 and
> 5 ...
>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
> Yahoo . yscrappy Skype: hub.org ICQ . 7615664
>

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

Marc,

>
> Please note ... at *no time* did we ever reject spam for any mailbox,
> except those for the mailing lists themselves, which majordomo is setup to
> do ... so if you are seeing more now then before, its because we are being
> hit with more now that are passing through spamassassin ...

Aha, ok, I see what happened. Some change you made disabled the spam filters
I had set up oh Horde. I need to re-enable them.

I need to find some way to set up a "default" spam configuration with all
regional contacts to make their accounts usable, which I guess would require
both settings in Maia and Horde. Suggestions?

> Not sure what you are finding awkward about the interface, but if you go
> in and just turn on all the various checks, but *disable* the
> quarantining, so that it just label's and passing everything over to you
> to filter, you will have exactly the same thing in place that was there
> before we added the interface for per-user settings ...

Got it. The interface isn't quite intuitive and the help file is unhelpful.
What I found awkward ... downright unusable, in fact ... is the quarantine
interface. Bad, bad webform. The fields are too small to display useful
data, it's not UTF-8, and there's no "check all". Of course, if you've been
using it, you know this.

What's Maia written in?

Question: does putting stuff in a "spam" folder via IMAP still work?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

On Mon, 7 Aug 2006, Josh Berkus wrote:

> Marc,
>
>
>>
>> Please note ... at *no time* did we ever reject spam for any mailbox,
>> except those for the mailing lists themselves, which majordomo is setup to
>> do ... so if you are seeing more now then before, its because we are being
>> hit with more now that are passing through spamassassin ...
>
> Aha, ok, I see what happened. Some change you made disabled the spam filters
> I had set up oh Horde. I need to re-enable them.
>
> I need to find some way to set up a "default" spam configuration with all
> regional contacts to make their accounts usable, which I guess would require
> both settings in Maia and Horde. Suggestions?

To be honest, since using MAIA, I think I've removed my own 'spam filters'
,,, with quarantine set for all messages scoring >5, those messages never
reach my filters, but stay in MAIA until I go in an report as spam, or
non-spam ...

So, in theory, it should just be a matter of enabling the appropriate
settings in MAIA for each of the regional contacts, and getting them to
login periodically to 'train' the system ...

If you *really* wanted the responsibility for it, I could link all of the
regional accounts to yours, which means when you login to 'train' the
system, you'd be training all of them ... your call on that one :)

> Got it. The interface isn't quite intuitive and the help file is
> unhelpful. What I found awkward ... downright unusable, in fact ... is
> the quarantine interface. Bad, bad webform. The fields are too small
> to display useful data, it's not UTF-8, and there's no "check all". Of
> course, if you've been using it, you know this.

'k, can't help with alot of the above ... but there is definitely a 'check
all' ... look at the very top line, before the first email is listed ...
each column as a 'checkbox', that if you click it, sets all boxes in that
column to checked ...

> What's Maia written in?

PHP ...

> Question: does putting stuff in a "spam" folder via IMAP still work?

Not if you are quarantining in MAIA, since it won't pass it to the
mailbox, but holds it in cache until you release it ... and its expected
that all you will release is, of course, non-SPAM ... the rest is
reported/trained and deleted from the web interface ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

Because all my momjian(at)postgresql(dot)org email is forwarded to
bruce(at)momjian(dot)us, I don't have any way of training the spam filter. Is
there a way I can use a global one?

---------------------------------------------------------------------------

Marc G. Fournier wrote:
> On Sat, 5 Aug 2006, Josh Berkus wrote:
>
> > Folks,
> >
> > Is there any way we can return to whatever we were using before Maia
> > mailgard? Maia is patently NOT working ... it's fundamentally broken in
> > some way, and isn't blocking any spam. The interface for Baysian
> > filtering is also awkward and darned near unusable. In a couple months
> > we're going to be trying to do the PostgreSQL 8.2 release which means
> > that we need the @postgresql.org e-mail addresses working ... and right
> > now they're all receiving a flood of unfiltered spam, making them
> > unusable.
>
> Actually, all MAIA is is a front-end over what we were using before
> (amavisd + spamassassin) to allow individual mailboxes to configure
> themselves, instead of one big site-wide setting ... I'm getting a
> *trickle* of spam right now, but I've spent the past week or so going
> through and training the Bayesian database ... in fact, under the old
> system, those 46 messages that are currently quarantined in your cache
> would have been delivered, instead of quarantined ... in the
> @postgresql.org one that I monitor, there are >7k messages currently in
> spam-quarantine that normally would have been passed through to the system
> ...
>
> Not sure what you are finding awkward about the interface, but if you go
> in and just turn on all the various checks, but *disable* the
> quarantining, so that it just label's and passing everything over to you
> to filter, you will have exactly the same thing in place that was there
> before we added the interface for per-user settings ...
>
> Please note ... at *no time* did we ever reject spam for any mailbox,
> except those for the mailing lists themselves, which majordomo is setup to
> do ... so if you are seeing more now then before, its because we are being
> hit with more now that are passing through spamassassin ...
>
> Note that unlike the ~60 messages in the cache for your mailbox right now
> (12 unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go
> through is right now sitting at ~10k, and I generally spend about a half
> hour in the evening going through it ... I generally care about
> unconfirmed-spam between 5 and 10, ignoring most above that ... and
> unconfirmed non-spam between 0 and 5 ...
>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
> Yahoo . yscrappy Skype: hub.org ICQ . 7615664
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Because all my momjian(at)postgresql(dot)org email is forwarded to
> bruce(at)momjian(dot)us, I don't have any way of training the spam filter. Is
> there a way I can use a global one?

I don't think that would work (except for maybe greylisting).

Sincerely,

Joshua D. Drake
--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Tue, 8 Aug 2006, Bruce Momjian wrote:

>
> Because all my momjian(at)postgresql(dot)org email is forwarded to
> bruce(at)momjian(dot)us, I don't have any way of training the spam filter. Is
> there a way I can use a global one?

MAIA is *before* your mailbox, so, before the filtering ... the mail
system essentially goes:

smtpd -> greylist -> maia -> mailbox -> local store
or
-> forward

so, you can still train the system, just login, go to Settings and set
your appropriate settings so that it Quarantines stuff, and then train
away ...

The nice thing about the system is that it doesn't just train/report to
the Bayes database, but it reports it to Razor2, Spamcop and Pyzor as
well, but, it does it in the background ... the interface just allows you
to confirm an email as spam, and a cron job then runs to actually do the
appropriate processing of the email(s) ...

any non-spam that you 'release' is, of course, released instantly back
into the system ...

there are a few things planned for v1.0.2 that will be nice ... like the
ability to just auto-discard anything that scores over X (if someone sends
me an email taht scores over, say, 20, chances are it isn't something I
want to read or ever really see) ...

>
> ---------------------------------------------------------------------------
>
> Marc G. Fournier wrote:
>> On Sat, 5 Aug 2006, Josh Berkus wrote:
>>
>>> Folks,
>>>
>>> Is there any way we can return to whatever we were using before Maia
>>> mailgard? Maia is patently NOT working ... it's fundamentally broken in
>>> some way, and isn't blocking any spam. The interface for Baysian
>>> filtering is also awkward and darned near unusable. In a couple months
>>> we're going to be trying to do the PostgreSQL 8.2 release which means
>>> that we need the @postgresql.org e-mail addresses working ... and right
>>> now they're all receiving a flood of unfiltered spam, making them
>>> unusable.
>>
>> Actually, all MAIA is is a front-end over what we were using before
>> (amavisd + spamassassin) to allow individual mailboxes to configure
>> themselves, instead of one big site-wide setting ... I'm getting a
>> *trickle* of spam right now, but I've spent the past week or so going
>> through and training the Bayesian database ... in fact, under the old
>> system, those 46 messages that are currently quarantined in your cache
>> would have been delivered, instead of quarantined ... in the
>> @postgresql.org one that I monitor, there are >7k messages currently in
>> spam-quarantine that normally would have been passed through to the system
>> ...
>>
>> Not sure what you are finding awkward about the interface, but if you go
>> in and just turn on all the various checks, but *disable* the
>> quarantining, so that it just label's and passing everything over to you
>> to filter, you will have exactly the same thing in place that was there
>> before we added the interface for per-user settings ...
>>
>> Please note ... at *no time* did we ever reject spam for any mailbox,
>> except those for the mailing lists themselves, which majordomo is setup to
>> do ... so if you are seeing more now then before, its because we are being
>> hit with more now that are passing through spamassassin ...
>>
>> Note that unlike the ~60 messages in the cache for your mailbox right now
>> (12 unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go
>> through is right now sitting at ~10k, and I generally spend about a half
>> hour in the evening going through it ... I generally care about
>> unconfirmed-spam between 5 and 10, ignoring most above that ... and
>> unconfirmed non-spam between 0 and 5 ...
>>
>> ----
>> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
>> Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
>> Yahoo . yscrappy Skype: hub.org ICQ . 7615664
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 6: explain analyze is your friend
>
> --
> Bruce Momjian bruce(at)momjian(dot)us
> EnterpriseDB http://www.enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +
>

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

OK, I set it up to forward, but not discard from my mailbox, and I will
use that for training. Good idea. Thanks.

---------------------------------------------------------------------------

Marc G. Fournier wrote:
> On Tue, 8 Aug 2006, Bruce Momjian wrote:
>
> >
> > Because all my momjian(at)postgresql(dot)org email is forwarded to
> > bruce(at)momjian(dot)us, I don't have any way of training the spam filter. Is
> > there a way I can use a global one?
>
> MAIA is *before* your mailbox, so, before the filtering ... the mail
> system essentially goes:
>
> smtpd -> greylist -> maia -> mailbox -> local store
> or
> -> forward
>
> so, you can still train the system, just login, go to Settings and set
> your appropriate settings so that it Quarantines stuff, and then train
> away ...
>
> The nice thing about the system is that it doesn't just train/report to
> the Bayes database, but it reports it to Razor2, Spamcop and Pyzor as
> well, but, it does it in the background ... the interface just allows you
> to confirm an email as spam, and a cron job then runs to actually do the
> appropriate processing of the email(s) ...
>
> any non-spam that you 'release' is, of course, released instantly back
> into the system ...
>
> there are a few things planned for v1.0.2 that will be nice ... like the
> ability to just auto-discard anything that scores over X (if someone sends
> me an email taht scores over, say, 20, chances are it isn't something I
> want to read or ever really see) ...
>
>
>
> >
> > ---------------------------------------------------------------------------
> >
> > Marc G. Fournier wrote:
> >> On Sat, 5 Aug 2006, Josh Berkus wrote:
> >>
> >>> Folks,
> >>>
> >>> Is there any way we can return to whatever we were using before Maia
> >>> mailgard? Maia is patently NOT working ... it's fundamentally broken in
> >>> some way, and isn't blocking any spam. The interface for Baysian
> >>> filtering is also awkward and darned near unusable. In a couple months
> >>> we're going to be trying to do the PostgreSQL 8.2 release which means
> >>> that we need the @postgresql.org e-mail addresses working ... and right
> >>> now they're all receiving a flood of unfiltered spam, making them
> >>> unusable.
> >>
> >> Actually, all MAIA is is a front-end over what we were using before
> >> (amavisd + spamassassin) to allow individual mailboxes to configure
> >> themselves, instead of one big site-wide setting ... I'm getting a
> >> *trickle* of spam right now, but I've spent the past week or so going
> >> through and training the Bayesian database ... in fact, under the old
> >> system, those 46 messages that are currently quarantined in your cache
> >> would have been delivered, instead of quarantined ... in the
> >> @postgresql.org one that I monitor, there are >7k messages currently in
> >> spam-quarantine that normally would have been passed through to the system
> >> ...
> >>
> >> Not sure what you are finding awkward about the interface, but if you go
> >> in and just turn on all the various checks, but *disable* the
> >> quarantining, so that it just label's and passing everything over to you
> >> to filter, you will have exactly the same thing in place that was there
> >> before we added the interface for per-user settings ...
> >>
> >> Please note ... at *no time* did we ever reject spam for any mailbox,
> >> except those for the mailing lists themselves, which majordomo is setup to
> >> do ... so if you are seeing more now then before, its because we are being
> >> hit with more now that are passing through spamassassin ...
> >>
> >> Note that unlike the ~60 messages in the cache for your mailbox right now
> >> (12 unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go
> >> through is right now sitting at ~10k, and I generally spend about a half
> >> hour in the evening going through it ... I generally care about
> >> unconfirmed-spam between 5 and 10, ignoring most above that ... and
> >> unconfirmed non-spam between 0 and 5 ...
> >>
> >> ----
> >> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> >> Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
> >> Yahoo . yscrappy Skype: hub.org ICQ . 7615664
> >>
> >> ---------------------------(end of broadcast)---------------------------
> >> TIP 6: explain analyze is your friend
> >
> > --
> > Bruce Momjian bruce(at)momjian(dot)us
> > EnterpriseDB http://www.enterprisedb.com
> >
> > + If your life is a hard drive, Christ can be your backup. +
> >
>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
> Yahoo . yscrappy Skype: hub.org ICQ . 7615664

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +