Re: logical changeset generation v6.2

On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
> On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> >> That strikes me as a flaw in the implementation rather than the idea.
> >> You're presupposing a patch where the necessary information is
> >> available in WAL yet you don't make use of it at the proper time.
> >
> > The problem is that the mapping would be somewhere *ahead* from the
> > transaction/WAL we're currently decoding. We'd need to read ahead till
> > we find the correct one.
>
> Yes, I think that's what you need to do.

My problem with that is that rewrite can be gigabytes into the future.

When reading forward we could either just continue reading data into the
reorderbuffer, but delay replaying all future commits till we found the
currently needed remap. That might have quite the additional
storage/memory cost, but runtime complexity should be the same as normal
decoding.
Or we could individually read ahead for every transaction. But doing so
for every transaction will get rather expensive (rougly O(amount_of_wal^2)).

I think that'd be pretty similar to just disallowing VACUUM
FREEZE/CLUSTER on catalog relations since effectively it'd be to
expensive to use.

> > But I think I mainly misunderstood what you proposed. That mapping could
> > be written besides relfilenode, instead of into the WAL. Then my
> > imagined problem doesn't exist anymore.
>
> That's pretty ugly. I think it should be written into WAL.

It basically has O(1) access, that's why I was thinking about it.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services