User control over psql error stream

Hi,

This patch gives the end user control over psql's
error stream. This allows a single psql session
to use \o to send both errors and table output
to multiple files. Useful when capturing test output, etc.

Control is provided via a new "estream" \pset. Here's
the docs.

-------------------------<snip>------------------------
estream

Controls the output stream(s) used to report error messages. Value
must be one of: stderr (the default), which sends errors to the
standard error stream; query, which injects error messages into the
query result output stream; or both, which sends errors to both output
streams. "Error messages" are comprised of errors from psql and notice
messages and errors from the database server.
-------------------------<snip>------------------------

Against head.

psql-estream.patch The patch.
psql-estream_test.patch Adds a regression test to test the patch.

There's a number of problems with psql-estream_test.patch,
the most notable is that it probably won't work on
MS Windows because it uses /dev/null to avoid touching the
host filesystem. I'm not sure whether this should have
a regression test and if so what the right way is to do it.

Note that psql-stream.patch includes some re-writing of
the docs for the psql \o option that goes slightly beyond
the minimum change required to explain \pset estream's effects.

Regards,

Karl <kop(at)meme(dot)com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein

On 11/15/12 3:53 PM, Karl O. Pinc wrote:
> This patch gives the end user control over psql's
> error stream. This allows a single psql session
> to use \o to send both errors and table output
> to multiple files. Useful when capturing test output, etc.

What does this do that cannot in practice be achieved using shell
redirection operators?

On 11/21/2012 01:41:56 PM, Peter Eisentraut wrote:
> On 11/15/12 3:53 PM, Karl O. Pinc wrote:
> > This patch gives the end user control over psql's
> > error stream. This allows a single psql session
> > to use \o to send both errors and table output
> > to multiple files. Useful when capturing test output, etc.
>
> What does this do that cannot in practice be achieved using shell
> redirection operators?

I've 3 answers to this. The short one, the long one,
and the practical one. Sorry to go on about it.

The brief generic answer is:

Easily capture error output associated with disparate
input streams, interspersed with the data output associated with
the disparate input streams, in separate files
correlated with the input, in the case where transactions
span multiple input streams. That latter
bit about transactions being the important part.

Of course I could be wrong/not clever enough.

The long generic answer is:

Shell redirection works fine if the transactions do not span
input streams; a separate connection/psql instance can
be used to redirect stderr intermixed with stdout as desired.
However when transactions span input streams then a single
psql instance/db connection is required to maintain
transaction state. In such a case it is difficult
to re-redirect stderr so that it's output appears
intermixed with table data output (stdout) in
time wise order while at the same time redirecting
all output into files that correlate with the
various input streams.

It's doable with something like:

psql 2>&1 < myfifo | mydemultiplexer &

where "myfifo" is a fifo fed the various input streams
and "mydemultiplexer" is a some process that
splits stdin into various output files as
various bits of input are sent to psql. But there
must be a way to communicate with the mydemultiplexer,
and you need to worry about synchronization of
input with output. And it's not clear to me how to
write mydemultiplexer in shell either.

Note that I've not actually tried the example above.

The practical answer is that I want my dependency
based test framework to work:

http://gombemi.ccas.gwu.edu/cgi-bin/darcsweb.cgi?r=gombemi;a=tree;f=/
db/test

For an intro see:

http://gombemi.ccas.gwu.edu/cgi-bin/darcsweb.cgi?
r=gombemi;a=headblob;f=/db/test/test.mk

I'm using this patch in my test framework.
If I had to I think I could get away
without it, but it would involve a bit of
work, and more than a bit of additional
complication.

Although I've got (much) further work to do
if I'm to follow through on my notion to
release this as a generic test framework,
what's documented in test.mk works. Probably.
(Not bad for under 300 lines of actual code,
comments excluded. Perhaps under 200 if you
eliminate blank lines and configuration for
my real-world use.)

Comments/suggestions are welcome.

Regards,

Karl <kop(at)meme(dot)com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein

On 11/21/2012 01:41:56 PM, Peter Eisentraut wrote:
> On 11/15/12 3:53 PM, Karl O. Pinc wrote:
> > This patch gives the end user control over psql's
> > error stream. This allows a single psql session
> > to use \o to send both errors and table output
> > to multiple files. Useful when capturing test output, etc.
>
> What does this do that cannot in practice be achieved using shell
> redirection operators?

To look at it from another angle, you need it for the
same reason you need \o -- to redirect output
to multiple places from within a single psql process.
\o redirects stdout, but I'm interested in
redirecting stderr.

Which makes me think that instead of a \pset
option a, say, \e, command is called for.
This would do the same thing \o does only
for stderr instead of stdout.

The problem comes
when you want \e to send to the same place \o
sends to, or vice versa. You can't just compare textual paths
because there's more than one way to write
a path to an object in the filesystem.
You need either a "special" argument
to \e and \o -- very ungood as it breaks
backwards compatibility with \o --
or you need, perhaps, yet another \
command to send stderr to where \o
is going or, conversely, send \o
to where stderr is going. Perhaps\oe
could send \o output to where stderr
is going at the moment and \eo could
send stderr output to where \o output
is going at the moment?

Sorry to ramble on. I'm tired.

Regards,

Karl <kop(at)meme(dot)com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein

On 11/21/2012 11:33:18 PM, Karl O. Pinc wrote:
> On 11/21/2012 01:41:56 PM, Peter Eisentraut wrote:
> > On 11/15/12 3:53 PM, Karl O. Pinc wrote:
> > > This patch gives the end user control over psql's
> > > error stream. This allows a single psql session
> > > to use \o to send both errors and table output
> > > to multiple files. Useful when capturing test output, etc.
> >
> > What does this do that cannot in practice be achieved using shell
> > redirection operators?
>
> To look at it from another angle, you need it for the
> same reason you need \o -- to redirect output
> to multiple places from within a single psql process.
> \o redirects stdout, but I'm interested in
> redirecting stderr.

There is an interaction here with \set EXIT_ON_ERROR
that may need additional documentation. As the patch is,
when \set EXIT_ON_ERROR is used the error stream is sent
to stderr in addition to the \pset estream setting.
At least I think this is what is happening.
This behavior seems correct to me.

Before proceeding further I would like to wait
for feedback regarding whether a \e command,
per above and previous, is preferable to the existing patch's
\pset estream.

Regards,

Karl <kop(at)meme(dot)com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein