From: | Hannu Krosing <hannu(at)2ndQuadrant(dot)com> |
---|---|
To: | Joshua Drake <jd(at)commandprompt(dot)com> |
Cc: | pgsql-hackers(at)postgresql(dot)org, Alvaro Herrera <alvherre(at)commandprompt(dot)com> |
Subject: | Re: PostgreSQL + Replicator developer meeting 10/28 |
Date: | 2008-10-28 23:45:04 |
Message-ID: | 1225237504.13402.33.camel@huvostro |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Tue, 2008-10-28 at 15:18 -0700, Joshua Drake wrote:
> On Tue, 28 Oct 2008 19:46:42 +0200
> Hannu Krosing <hannu(at)2ndQuadrant(dot)com> wrote:
>
> > > The current topics are:
> > >
> > > * New MCP architecture
> >
> > What's new ?
> >
> > I have some doubts about the current architecture based on my reading
> > of replicator wiki, but would like to learn about the "new"
> > architecture before spending too much time in studying the "old" one.
>
> The two obvious problems with the existing MCP architecture is:
>
> 1. Single point of failure
For async replication there is always SPoF, at least the master until
first slave has aquired log is a SPoF, or do you plan that both Master
and "MCP|Slave" to keep the log and be able to step in for each other if
the other fails?
> 2. Portability
Portability to where ? Other DBMS's ? Other PG versions ?
for me there was also two more problems:
3. separate "replication log", which at least seems to be able to get
out of sync with main DB.
Why don't you just use a DB table, WAL-logged and all
4. Also, again from reading Replicator FAQ, it seems that there is a
window of corruption/data loss when rotating the Replicators transaction
log. I think that doing it with copy/truncate either needs locking the
logfile (== bad performance, during copy/truncate) or is just a
data-eating failure waiting to happen.
pgQ has a solution to that by rotating queue tables, and postgreSQL core
also does effectively "rotate" WAL log segments. To be robust _and_
effective, Replicator should also rotate the logfile itself.
> The new architecture is set to remove both of those. The short version
> is the MCP will be moved into the backend. Thus:
>
> Master->MCP|Slave ->Slave1
> ->Slave2
> ->Slave3
>
> The process being, Master sends data to MCP|Slave, MCP|Slave writes it
> to disk (optionally restores it)
Will this first send be sync or async ? Or have you planned to have it
be configurable among several robustness vs. performance levels, similar
to the planned integrated WAL-shipping.
if async, will it also use MVCC for keeping log on Master (l.ike Slony
and pgQ do), just to be at least as reliable as postgreSQL core itself
and not require a full resync at server crash.
> and then forwards it to 1,2,3 who then
> receive the data, write it to disk and then restore it.
>
> If master dies, you can promote to any of the slaves and the left over
> slaves will connect to the promoted slave and begin receiving updates.
>
> If the MCP|Slave dies a new Slave can begin the MCP|Slave process.
>
> Alvaro or Alexey can speak more technically about implementation than I
> can.
Alvaro - I guess you already have discussed most of it, but basically
you need to solve all the same problems that WAL-shipping based Hot
Standby is solving and Slony/pgQ/Londiste have solved.
Hopefully you get it more robust than Slony when making changes under
high load :)
Will there be an helper application for setting up and configuring
changes in replication. or will it all be done using added SQL
commands ?
How will DDL be handled ( i understood that you don't yet have DDL
replication) ?
Will Slave tables be kind-of-read-only like Slony slaves ? Or even
_really_ read only like Simon's Hot Standby ?
> >
> > > * DDL Replication
> >
> > Is it there alread, or is it just a planned feature ?
>
> Planned feature.
Did the plans got any clearer during this meeting ?
> > * How hard would it be to extract DDL replication part and use it as
> > basis for DDL after trigger support for use in trigger based
> > replication/auditing like Slony ann pgQ/Londiste ?
>
> Hmm I am not sure. We are pretty deep into the core and only use
> triggers for GRANT/REVOKE/CREATE ROLE .
By the way, why did you choose "pretty deep into the core" approach
instead of triggers ?
I mean, you probably end up duplicating (or missing) lots of
postgreSQL-s internal goodness instead of just using what is already
available ?
> Sincerely,
>
> Joshua D. Drake
Thanks for the update.
I hope something useful will come out of this too, though I hoped that
it already had some advantages over trigger-based replication, like
ability to replicate DDL .
-----------------------
Hannu Krosing
From | Date | Subject | |
---|---|---|---|
Next Message | Gregory Stark | 2008-10-28 23:54:21 | Re: Proposal of PITR performance improvement for 8.4. |
Previous Message | Bruce McAlister | 2008-10-28 23:01:33 | UUID-OSSP Contrib Module Compilation Issue |