Sketch of a Hook into the Logging Collector

Hello all,

I have some needs that seem to support changing Postgres slightly to
give user programs a lot more power over how to process logging output
that neither the log collector nor the syslog output can well-satisfy
as-is.

I am approaching this from the angle of increasing power by exposing
the log collector ("syslogger") pipe protocol.

Included is a patch whose general aesthetic is to touch as little of
the logging code as possible while achieving maximum power in a
contrib and stand-alone sample program. However, a more satisfying
treatment that may be slightly more invasive to the logging system is
very welcome.

The general idea here is that the logging collector pipe protocol is a
pretty reasonable one for non-Postgres programs to consume as-is
pretty easily. The proof of concept consists of three things:

* A hook into the log collector. It is small but a little weird
compared to the other hooks in the system.

* A contrib, pg_logcollectdup. This contrib lets one forward logs to
a named pipe specified in postgresql.conf.

* A stand-alone program, pg_logcollect_sample, that renders protocol
traffic newline separated and with its binary fields converted to
readable text.

The patch can also be seen from https://github.com/fdr/postgres.git in
the branch 'logdup'. I do rebase this at-will for the time being.

I have a few detailed dissatisfactions with this approach, but I'd
rather hear your own dissatisfactions. I also don't like the name.

A demo of configuring all of the above and seeing some output follows.
I can paste this in one go on my machine. It creates /tmp/pg-logdup
and /tmp/pgdata-logdup. All spawned processes can be listed
afterwards with 'jobs'.

# Install postgres and pg_logcollectdup contrib
./configure --prefix=/tmp/pg-logdup
make -sj16 install
cd contrib/pg_logcollectdup/
make -s install
/tmp/pg-logdup/bin/initdb -D /tmp/pgdata-logdup
mkfifo /tmp/pgdata-logdup/log-pipe

# Configure postgresql.conf

echo "logging_collector = on" >> \
/tmp/pgdata-logdup/postgresql.conf

echo "shared_preload_libraries = 'pg_logcollectdup'" >> \
/tmp/pgdata-logdup/postgresql.conf

echo "logcollectdup.destination = '/tmp/pgdata-logdup/log-pipe'" >> \
/tmp/pgdata-logdup/postgresql.conf

# Set up destination pipe
mkfifo /tmp/pgdata-logdup/log-pipe

# Build sample pipe formatter
make pg_logcollect_sample

# Run it in the background
./pg_logcollect_sample \
< /tmp/pgdata-logdup/log-pipe \
> /tmp/pgdata-logdup/duplogs.txt &

# Run Postgres with a non-default port for convenience
/tmp/pg-logdup/bin/postgres -D /tmp/pgdata-logdup --port=2345 &

# Prevent race against file creation, then look at the logs
sleep 1
cat /tmp/pgdata-logdup/duplogs.txt

--
fdr