improving log management

Hi,

We support several methods for logging server messages. The "native" methods
(stderr, csvlogs) has poor management. We can't compress logs, send them to
another location/server, or just remove old ones. Another problem is that we
can't remove (automatically) old logs based on the number of existing log
files (of course we can do it using log_truncate_on_rotation and a
log_filename that matches the number of files -- but it *isn't* flexible). The
other log management softwares have a way to do that so why our logger doesn't
have such capability?

I want to propose a new command to be execute after the log file is rotated. A
GUC parameter log_after_rotation_command that takes a (set of) command(s) that
will be executed after a log file is rotated. It should be set only by
superuser. For example:

# compress and store the log file in another location
log_after_rotation_command = 'gzip %f && mv %p.gz /a/mylogs'

Depending on the comments, I will clean up the patch and send it later today.

--
Euler Taveira de Oliveira
http://www.timbira.com/

Euler Taveira de Oliveira <euler(at)timbira(dot)com> writes:
> I want to propose a new command to be execute after the log file is
> rotated.

(1) Windows compatibility?

(2) What happens if the command takes a significant amount of time to
execute? We can't afford to have the log collector blocked.

(3) What do you intend those %p and %f symbols to signify?

regards, tom lane

Euler Taveira de Oliveira <euler(at)timbira(dot)com> wrote:

> other log management softwares have a way to do that so why our logger doesn't
> have such capability?
>
> I want to propose a new command to be execute after the log file is rotated. A
> GUC parameter log_after_rotation_command that takes a (set of) command(s) that
> will be executed after a log file is rotated.

If you have better loggers already, why don't you use them? In another word,
should we cooperate with them instead of re-inventing alternative loggers?

We have "Logging Brainstorm" topic in out wiki. It might help you.
http://wiki.postgresql.org/wiki/Logging_Brainstorm

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center

Tom Lane escreveu:
> (1) Windows compatibility?
>
Yes.

> (2) What happens if the command takes a significant amount of time to
> execute? We can't afford to have the log collector blocked.
>
You're right. We can't have that command in the same process because DBAs
could have high values for log_rotation_* parameters or resources could become
unavailable. :( One idea is to have another process (logger worker) to execute
it. Also we need to store the logfiles that were already rotated.

> (3) What do you intend those %p and %f symbols to signify?
>
%p and %f are relative log path and log filename, respectively.

--
Euler Taveira de Oliveira
http://www.timbira.com/

Takahiro Itagaki escreveu:
> If you have better loggers already, why don't you use them? In another word,
> should we cooperate with them instead of re-inventing alternative loggers?
>
We can use but they don't features that our logger has. If we have a logger,
let's improve it.

> We have "Logging Brainstorm" topic in out wiki. It might help you.
> http://wiki.postgresql.org/wiki/Logging_Brainstorm
>
Interesting. Don't know about that. Why some of those ideas aren't in our TODO?

--
Euler Taveira de Oliveira
http://www.timbira.com/