Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

On Monday, September 17, 2012 01:50:33 PM Heikki Linnakangas wrote:
> On 17.09.2012 14:42, Andres Freund wrote:
> > On Monday, September 17, 2012 12:55:47 PM Heikki Linnakangas wrote:
> >> On 17.09.2012 13:01, Andres Freund wrote:
> >>> On Monday, September 17, 2012 11:07:28 AM Andres Freund wrote:
> >>>> On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote:
> >>>>> On 17.09.2012 11:12, Andres Freund wrote:
> >>>>>> On Monday, September 17, 2012 09:40:17 AM Heikki Linnakangas wrote:
> >>>>>> If you don't want the capability to forward/filter the data and read
> >>>>>> partial data without regard for record constraints/buffering your
> >>>>>> patch seems to be quite a good start. It misses xlogreader.h
> >>>>>> though...
> >>>>>
> >>>>> Ah sorry, patch with xlogreader.h attached.
> >>>>
> >>>> Will look at it in a second.
> >>>
> >>> It seems we would need one additional callback for both approaches
> >>> like:
> >>>
> >>> ->error(severity, format, ...)
> >>>
> >>> For both to avoid having to draw in elog.c.
> >>
> >> Yeah. Another approach would be to return the error string from
> >> ReadRecord. The caller could then do whatever it pleases with it, like
> >> ereport() it to the log or PANIC. I think I'd like that better.
> >
> > That seems a bit more complex from a memory management perspective as you
> > probably would have to sprintf() into some buffer. We cannot rely on a
> > backend environment with memory contexts around et al...
>
> Hmm. I was thinking that making this work in a non-backend context would
> be too hard, so I didn't give that much thought, but I guess there isn't
> many dependencies to backend functions after all. palloc/pfree are
> straightforward to replace with malloc/free.
Hm. I thought that it was pretty much a design requirement that this is usable
outside of the backend environment?

> That's what we could easily do with the error messages too, just malloc a
> suitably sized buffer.
Not very comfortable though... Especially if you need to return an error from
the read_page callback...

> How does a non-backend program get access to xlogreader.c? Copy
> xlogreader.c from the source tree at build time and link into the
> program? Or should we turn it into a shared library?
Not really sure. I thought about just putting it in pgport or such, but that
seemed ugly as well.
The bin/xlogdump hack, which I find really helpful, at first simply had a
dependency on ../../backend/access/transam/xlogreader.o which worked fine. Till
it needed more because of *_desc routines... But Alvaro started to work on this
although I don't know when he will be able to finish it.

Greetings,

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services