Re: logical changeset generation v4

On 13-01-14 08:38 PM, Andres Freund wrote:
> Hi everyone,
>
> Here is the newest version of logical changeset generation.
>
>
>
> 2) Currently the logical replication infrastructure assigns a 'slot-id'
> when a new replica is setup. That slot id isn't really nice
> (e.g. "id-321578-3"). It also requires that [18] keeps state in a global
> variable to make writing regression tests easy.
>
> I think it would be better to make the user specify those replication
> slot ids, but I am not really sure about it.

Shortly after trying out the latest version I hit the following scenario
1. I started pg_receivellog but mistyped the name of my plugin
2. It looped and used up all of my logical replication slots

I killed pg_receivellog and restarted it with the correct plugin name
but it won't do anything because I have no free slots. I can't free the
slots with -F because I have no clue what the names of the slots are.
I can figure the names out by looking in pg_llog but if my replication
program can't do that so it won't be able to clean up from a failed attempt.

I agree with you that we should make the user program specify a slot, we
eventually might want to provide a view that shows the currently
allocated slots. For a logical based slony I would just generate the
slot name based on the remote node id. If walsender generates the slot
name then I would need to store a mapping between slot names and slons
so when a slon restarted it would know which slot to resume using. I'd
have to use a table in the slony schema on the remote database for
this. There would always be a risk of losing track of a slot id if the
slon crashed after getting the slot number but before committing the
mapping on the remote database.

> 3) Currently no options can be passed to an output plugin. I am thinking
> about making "INIT_LOGICAL_REPLICATION 'plugin'" accept the now widely
> used ('option' ['value'], ...) syntax and pass that to the output
> plugin's initialization function.

I think we discussed this last CF, I like this idea.

> 4) Does anybody object to:
> -- allocate a permanent replication slot
> INIT_LOGICAL_REPLICATION 'plugin' 'slotname' (options);
>
> -- stream data
> START_LOGICAL_REPLICATION 'slotname' 'recptr';
>
> -- deallocate a permanent replication slot
> FREE_LOGICAL_REPLICATION 'slotname';

+1

> 5) Currently its only allowed to access catalog tables, its fairly
> trivial to extend this to additional tables if you can accept some
> (noticeable but not too big) overhead for modifications on those tables.
>
> I was thinking of making that an option for tables, that would be useful
> for replication solutions configuration tables.

I think this will make the life of anyone developing a new replication
system easier. Slony has a lot of infrastructure for allowing slonik
scripts to wait for configuration changes to popogate everywhere before
making other configuration changes because you can get race conditions.
If I were designing a new replication system and I had this feature then
I would try to use it to come up with a simpler model of propagating
configuration changes.

>
> Andres Freund
>
>

Steve