Re: logical changeset generation v6.2

On 2013-10-14 09:36:03 -0400, Robert Haas wrote:
> On Fri, Oct 11, 2013 at 12:57 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> >> I don't see any need for SQL syntax. I was just thinking that the
> >> _PG_init function could fill in a structure and then call
> >> RegisterLogicalReplicationOutputPlugin(&mystruct).
> >
> > Hm. We can do that, but what'd be the advantage of that? The current
> > model will correctly handle things like a'shared_preload_libraries'ed
> > output plugin, because its _PG_init() will not register it. With the
> > handling done in _PG_init() there would be two.
> > Being able to use the same .so for output plugin handling and some other
> > replication solution specific stuff is imo useful.
>
> Well, I just think relying on specific symbol names in the .so file is
> kind of unfortunate. It means that, for example, you can't have
> multiple output plugins provided by a single .so. And in general I
> think it's something that we've tried to minimize.

But that's not really different when you rely on _PG_init doing it's
thing, right?

> >> I don't see why you're so pessimistic about that. I know you haven't
> >> worked it out yet, but what makes this harder than sitting down and
> >> designing something?
> >
> > Because every replication solution has different requirements for the
> > format and they will want filter the output stream with regard to their
> > own configuration.
> > E.g. bucardo will want to include the transaction timestamp for conflict
> > resolution and such.
>
> But there's only so much information available here. Why not just
> have a format that logs it all?

Because we do not know what "all" is? Also, how would we handle
replication sets and such that all of the existing replication solutions
have generically?

> > Sure, that's no problem. Do I understand correctly that you'd like
> > wal_decoding: Add information about a tables primary key to struct RelationData
> > wal_decoding: Add wal_level = logical and log data required for logical decoding
> >
> > earlier?
>
> Yes.

That's done. Hope the new order makes sense.

> > So, if we're decoding data that needs to lookup those rows in TX1 or TX2
> > we both times need access to cmin and cmax, but neither transaction will
> > have created a multixact. That can only be an issue in transaction with
> > catalog modifications.

> Oh, yuck. So that means you have to write an extra WAL record for
> EVERY heap insert, update, or delete to a catalog table? OUCH.

Yes. We could integrate it into the main record without too many
problems, but it didn't seem like an important optimization and it would
have higher chances of slowing down wal_level < logical.

> > It'd probably not hurt to redo those benchmarks to make sure...
>
> Yes, I think it would be good to characterize it more precisely than
> "a bit", so people know what to expect.

A "bit" was below the 3% range for loops of adding columns.

So, any tests you'd like to see?
* loop around CREATE TABLE/DROP TABLE
* loop around ALTER TABLE ... ADD COLUMN
* loop around CREATE FUNCTION/DROP FUNCTION

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services