xlogdump

There's an xlogdump project on pgfoundry. However it suffers from perennial
bitrot as it has to maintain its own table of xlog record types and code to
decode each xlog record type.

Earlier I modified xlogdump to generate a CSV loadable data set so I could do
some basic analysis and see what types of operations are generating the most
wal traffic. But I found it had bitrotted and needed some attention to bring
it up to date.

Again now I wanted to repeat that analysis to measure the effect HOT has had
on WAL traffic. And again now I find it has bitrotted, not least because of
HOT of course...

I think this module should be rewritten to depend more closely on the Postgres
source files. What I'm doing now is making an SRF in the style of the
pageinspect module which will read an arbitrary wal file and generate records
directly.

This has a big disadvantage compared to the original approach, namely that you
need a functioning Postgres instance of the same version to dissect wal
records.

But it also has a big advantage, namely that it will always be in sync. It
will just use the same RmgrTable to find the rm_name and call the rm_desc
method to decode the record. The result might not be quite as or dense as the
rm_desc function is meant for debugging messages. We could address that
sometime with a new method if we wanted to.

I'm thinking of actually dropping it directly into the pageinspect contrib
module. It's not quite an exact fit but it doesn't seem to deserve it's own
contrib module and it's likely to suffer the same bitrot problem if it lives
in pgfoundry.

Incidentally I would like to call xlog.c:RecordIsValid() which is currently a
static function. Any objection to exporting it? It doesn't depend on any
external xlog.c state.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

Gregory Stark wrote:
> There's an xlogdump project on pgfoundry. However it suffers from perennial
> bitrot as it has to maintain its own table of xlog record types and code to
> decode each xlog record type.
>
> ...
>
> I think this module should be rewritten to depend more closely on the Postgres
> source files. What I'm doing now is making an SRF in the style of the
> pageinspect module which will read an arbitrary wal file and generate records
> directly.
>
> This has a big disadvantage compared to the original approach, namely that you
> need a functioning Postgres instance of the same version to dissect wal
> records.
>
> But it also has a big advantage, namely that it will always be in sync. It
> will just use the same RmgrTable to find the rm_name and call the rm_desc
> method to decode the record. The result might not be quite as or dense as the
> rm_desc function is meant for debugging messages. We could address that
> sometime with a new method if we wanted to.

Would it still be possible to compile it as a stand-alone program, using
the backend source files? It would be a hack, we just went through some
effort to clean up references to server private header files from ecpg
and initdb, but it feels a lot nicer to use as a standalone program than
requiring a running postgres instance.

How much infrastructure would you need to call rm_name and rm_desc from
a standalone program? palloc and friends, I presume, What else?

> I'm thinking of actually dropping it directly into the pageinspect contrib
> module. It's not quite an exact fit but it doesn't seem to deserve it's own
> contrib module and it's likely to suffer the same bitrot problem if it lives
> in pgfoundry.

I'd vote for pgfoundry or a new contrib module. It shouldn't suffer from
bitrot as easily as what's there now. That was the whole point of
switching over to the new approach, right?

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas wrote:
> Gregory Stark wrote:
>> There's an xlogdump project on pgfoundry. However it suffers from
>> perennial
>> bitrot as it has to maintain its own table of xlog record types and
>> code to
>> decode each xlog record type.
>>
>> ...
>>
>> I think this module should be rewritten to depend more closely on the
>> Postgres
>> source files. What I'm doing now is making an SRF in the style of the
>> pageinspect module which will read an arbitrary wal file and generate
>> records
>> directly.
>>
>> This has a big disadvantage compared to the original approach, namely
>> that you
>> need a functioning Postgres instance of the same version to dissect wal
>> records.
>>
>> But it also has a big advantage, namely that it will always be in
>> sync. It
>> will just use the same RmgrTable to find the rm_name and call the rm_desc
>> method to decode the record. The result might not be quite as or dense
>> as the
>> rm_desc function is meant for debugging messages. We could address that
>> sometime with a new method if we wanted to.
>
> Would it still be possible to compile it as a stand-alone program, using
> the backend source files? It would be a hack, we just went through some
> effort to clean up references to server private header files from ecpg
> and initdb, but it feels a lot nicer to use as a standalone program than
> requiring a running postgres instance.
>
> How much infrastructure would you need to call rm_name and rm_desc from
> a standalone program? palloc and friends, I presume, What else?

Do you actually need palloc and friends, or just "something named
palloc"? We already have some stuff in src/port that deals with using
palloc calls in routines used in frontend programs...

//Magnus

--On Freitag, November 02, 2007 10:54:45 +0000 Gregory Stark
<stark(at)enterprisedb(dot)com> wrote:

> I think this module should be rewritten to depend more closely on the
> Postgres source files. What I'm doing now is making an SRF in the style
> of the pageinspect module which will read an arbitrary wal file and
> generate records directly.

Isn't that already partially implemented in xlogviewer, hosted on the same
project:

<http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/xlogviewer/xlogviewer/

The last time i've used it it had some problems with exhausted memory, but
it implements a SRF returning WAL records from a given log segment file.

--
Thanks

Bernd

On Fri, 2007-11-02 at 10:54 +0000, Gregory Stark wrote:

> Incidentally I would like to call xlog.c:RecordIsValid() which is currently a
> static function. Any objection to exporting it? It doesn't depend on any
> external xlog.c state.

You'll have some fun with that because most of the stuff in xlog.c
depends upon module-level state.

We should change that, but I'd like to see Florian's code go in first
then refactor things.

If you change those programs too much we'll need completely separate
programs for each release, which might be a bad thing.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Added to TODO:

* Move pgfoundry's xlogdump to /contrib and have it rely more closely
on the WAL backend code

http://archives.postgresql.org/pgsql-hackers/2007-11/msg00035.php

---------------------------------------------------------------------------

Gregory Stark wrote:
>
> There's an xlogdump project on pgfoundry. However it suffers from perennial
> bitrot as it has to maintain its own table of xlog record types and code to
> decode each xlog record type.
>
> Earlier I modified xlogdump to generate a CSV loadable data set so I could do
> some basic analysis and see what types of operations are generating the most
> wal traffic. But I found it had bitrotted and needed some attention to bring
> it up to date.
>
> Again now I wanted to repeat that analysis to measure the effect HOT has had
> on WAL traffic. And again now I find it has bitrotted, not least because of
> HOT of course...
>
> I think this module should be rewritten to depend more closely on the Postgres
> source files. What I'm doing now is making an SRF in the style of the
> pageinspect module which will read an arbitrary wal file and generate records
> directly.
>
> This has a big disadvantage compared to the original approach, namely that you
> need a functioning Postgres instance of the same version to dissect wal
> records.
>
> But it also has a big advantage, namely that it will always be in sync. It
> will just use the same RmgrTable to find the rm_name and call the rm_desc
> method to decode the record. The result might not be quite as or dense as the
> rm_desc function is meant for debugging messages. We could address that
> sometime with a new method if we wanted to.
>
> I'm thinking of actually dropping it directly into the pageinspect contrib
> module. It's not quite an exact fit but it doesn't seem to deserve it's own
> contrib module and it's likely to suffer the same bitrot problem if it lives
> in pgfoundry.
>
> Incidentally I would like to call xlog.c:RecordIsValid() which is currently a
> static function. Any objection to exporting it? It doesn't depend on any
> external xlog.c state.
>
> --
> Gregory Stark
> EnterpriseDB http://www.enterprisedb.com
> Ask me about EnterpriseDB's On-Demand Production Tuning
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +