Logical replication - TRAP: FailedAssertion in pgstat.c

Testing logical replication, with the following patches on top of
yesterday's master:

0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patch

Is applying that patch set is still correct?

It builds fine, but when I run the old pbench-over-logical-replication
test I get:

TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)

reliably (often within a minute).

The test itself does not fail, at least not that I saw (but I only ran a
few).

thanks,

Erik Rijkers

On 2017-04-15 04:47, Erik Rijkers wrote:
>
> 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
> 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
> 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
> 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
> 0005-Skip-unnecessary-snapshot-builds.patch

I am now using these newer patches:
https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com

> It builds fine, but when I run the old pbench-over-logical-replication
> test I get:
>
> TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
> "pgstat.c", Line: 828)

To get that error:

--------------
#!/bin/sh

port1=6972 port2=6973 scale=25 clients=16 duration=60

echo "drop table if exists pgbench_accounts;
drop table if exists pgbench_branches;
drop table if exists pgbench_tellers;
drop table if exists pgbench_history;" | psql -qXp $port1 \
&& echo "drop table if exists pgbench_accounts;
drop table if exists pgbench_branches;
drop table if exists pgbench_tellers;
drop table if exists pgbench_history;" | psql -qXp $port2 \
&& pgbench -p $port1 -qis ${scale//_/} && echo "
alter table pgbench_history add column hid serial primary key;
" | psql -q1Xp $port1 \
&& pg_dump -F c -p $port1 \
--exclude-table-data=pgbench_history \
--exclude-table-data=pgbench_accounts \
--exclude-table-data=pgbench_branches \
--exclude-table-data=pgbench_tellers \
-t pgbench_history \
-t pgbench_accounts \
-t pgbench_branches \
-t pgbench_tellers \
| pg_restore -1 -p $port2 -d testdb

appname=pgbench_derail
echo "create publication pub1 for all tables;" | psql -p $port1 -aqtAX
echo "create subscription sub1 connection 'port=${port1}
application_name=${appname}' publication pub1 with (disabled);
alter subscription sub1 enable;
" | psql -p $port2 -aqtAX

echo "-- pgbench -p $port1 -c $clients -T $duration -n -- scale $scale
"
pgbench -p $port1 -c $clients -T $duration -n

--------------

Erik Rijkers

On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
> On 2017-04-15 04:47, Erik Rijkers wrote:
> >
> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
> > 0005-Skip-unnecessary-snapshot-builds.patch
>
> I am now using these newer patches:
> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>
> > It builds fine, but when I run the old pbench-over-logical-replication
> > test I get:
> >
> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
> > "pgstat.c", Line: 828)
>
>
> To get that error:

I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
if you git revert that individual commit, do things work again?

- Andres

On 16/04/17 20:41, Andres Freund wrote:
> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>> On 2017-04-15 04:47, Erik Rijkers wrote:
>>>
>>> 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>>> 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>>> 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>>> 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>>> 0005-Skip-unnecessary-snapshot-builds.patch
>>
>> I am now using these newer patches:
>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>
>>> It builds fine, but when I run the old pbench-over-logical-replication
>>> test I get:
>>>
>>> TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>>> "pgstat.c", Line: 828)
>>
>>
>> To get that error:
>
> I presume this is the fault of
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
> if you git revert that individual commit, do things work again?
>
> - Andres
>

Yeah it is, it needs to be fenced to happen only after commit, which is
not guaranteed at the point of code, we probably need to put the
pgstat_report_stat() inside the if above after the
CommitTransactionCommand() (that will make it report stats for changes
apply did to pg_subscription_rel after next transaction though) or put
it into it's own if that checks if tx was indeed committed (which is bit
ugly).

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2017-04-16 20:41, Andres Freund wrote:
> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>> On 2017-04-15 04:47, Erik Rijkers wrote:
>> >
>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>> > 0005-Skip-unnecessary-snapshot-builds.patch
>>
>> I am now using these newer patches:
>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>
>> > It builds fine, but when I run the old pbench-over-logical-replication
>> > test I get:
>> >
>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>> > "pgstat.c", Line: 828)
>>
>>
>> To get that error:
>
> I presume this is the fault of
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
> if you git revert that individual commit, do things work again?
>

Yes, compiled from 67c2def11d4 with the above 4 patches, it runs
flawlessly again. (flawlessly= a few hours without any error)

> On 17 Apr 2017, at 10:30, Erik Rijkers <er(at)xs4all(dot)nl> wrote:
>
> On 2017-04-16 20:41, Andres Freund wrote:
>> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>>> On 2017-04-15 04:47, Erik Rijkers wrote:
>>> >
>>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>>> > 0005-Skip-unnecessary-snapshot-builds.patch
>>> I am now using these newer patches:
>>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>> > It builds fine, but when I run the old pbench-over-logical-replication
>>> > test I get:
>>> >
>>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>>> > "pgstat.c", Line: 828)
>>> To get that error:
>> I presume this is the fault of
>> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
>> if you git revert that individual commit, do things work again?
>
> Yes, compiled from 67c2def11d4 with the above 4 patches, it runs flawlessly again. (flawlessly= a few hours without any error)
>

I’ve reproduced failure, this happens under tablesync worker and putting
pgstat_report_stat() under the previous condition block should help.

However for me it took about an hour of running this script to catch original assert.

Can you check with that patch applied?

On 2017-04-17 15:59, Stas Kelvich wrote:
>> On 17 Apr 2017, at 10:30, Erik Rijkers <er(at)xs4all(dot)nl> wrote:
>>
>> On 2017-04-16 20:41, Andres Freund wrote:
>>> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
>>>> On 2017-04-15 04:47, Erik Rijkers wrote:
>>>> >
>>>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
>>>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
>>>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
>>>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
>>>> > 0005-Skip-unnecessary-snapshot-builds.patch
>>>> I am now using these newer patches:
>>>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com
>>>> > It builds fine, but when I run the old pbench-over-logical-replication
>>>> > test I get:
>>>> >
>>>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
>>>> > "pgstat.c", Line: 828)
>>>> To get that error:
>>> I presume this is the fault of
>>> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
>>> if you git revert that individual commit, do things work again?
>>
>> Yes, compiled from 67c2def11d4 with the above 4 patches, it runs
>> flawlessly again. (flawlessly= a few hours without any error)
>>
>
> I’ve reproduced failure, this happens under tablesync worker and
> putting
> pgstat_report_stat() under the previous condition block should help.
>
> However for me it took about an hour of running this script to catch
> original assert.
>
> Can you check with that patch applied?

Your patch on top of the 5 patches above seem to solve the matter too:
no problems after running for 2 hours (previously it failed within half
a minute).

Erik Rijkers

On 4/16/17 16:11, Petr Jelinek wrote:
> Yeah it is, it needs to be fenced to happen only after commit, which is
> not guaranteed at the point of code, we probably need to put the
> pgstat_report_stat() inside the if above after the
> CommitTransactionCommand() (that will make it report stats for changes
> apply did to pg_subscription_rel after next transaction though)

I think to avoid the latter, we should add more pgstat_report_stat()
calls, such as in process_syncing_tables_for_apply(). Basically every
code path that calls CommitTransactionCommand() should have one, no?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> On 4/16/17 16:11, Petr Jelinek wrote:
>> Yeah it is, it needs to be fenced to happen only after commit, which is
>> not guaranteed at the point of code, we probably need to put the
>> pgstat_report_stat() inside the if above after the
>> CommitTransactionCommand() (that will make it report stats for changes
>> apply did to pg_subscription_rel after next transaction though)
>
> I think to avoid the latter, we should add more pgstat_report_stat()
> calls, such as in process_syncing_tables_for_apply(). Basically every
> code path that calls CommitTransactionCommand() should have one, no?

Is there anything left to be committed here?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 02/05/17 20:43, Robert Haas wrote:
> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
> <peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
>> On 4/16/17 16:11, Petr Jelinek wrote:
>>> Yeah it is, it needs to be fenced to happen only after commit, which is
>>> not guaranteed at the point of code, we probably need to put the
>>> pgstat_report_stat() inside the if above after the
>>> CommitTransactionCommand() (that will make it report stats for changes
>>> apply did to pg_subscription_rel after next transaction though)
>>
>> I think to avoid the latter, we should add more pgstat_report_stat()
>> calls, such as in process_syncing_tables_for_apply(). Basically every
>> code path that calls CommitTransactionCommand() should have one, no?
>
> Is there anything left to be committed here?
>

Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2017-05-03 08:17, Petr Jelinek wrote:
> On 02/05/17 20:43, Robert Haas wrote:
>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut

>>> code path that calls CommitTransactionCommand() should have one, no?
>>
>> Is there anything left to be committed here?
>>
>
> Afaics the fix was not committed. Peter wanted more comprehensive fix
> which didn't happen. I think something like attached should do the job.

I'm running my pgbench-over-logical-replication test in chunk of 15
minutes, wth different pgbench -c (num clients) and -s (scale) values.

With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):

> fix-statistics-reporting-in-logical-replication-work.patch

logical replication is still often failing (as expected, I suppose; it
seems because of "inital snapshot too large") but indeed I do not see
the 'TRAP: FailedAssertion in pgstat.c' anymore.

(If there is any other configuration of patches worth testing please let
me know)

thanks

Erik Rijkers

On Wed, May 03, 2017 at 01:23:19PM +0200, Erik Rijkers wrote:
> On 2017-05-03 08:17, Petr Jelinek wrote:
> >On 02/05/17 20:43, Robert Haas wrote:
> >>On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
>
> >>>code path that calls CommitTransactionCommand() should have one, no?
> >>
> >>Is there anything left to be committed here?
> >>
> >
> >Afaics the fix was not committed. Peter wanted more comprehensive fix
> >which didn't happen. I think something like attached should do the job.
>
> I'm running my pgbench-over-logical-replication test in chunk of 15 minutes,
> wth different pgbench -c (num clients) and -s (scale) values.
>
> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
> precise):
>
> >fix-statistics-reporting-in-logical-replication-work.patch
>
> logical replication is still often failing (as expected, I suppose; it seems
> because of "inital snapshot too large") but indeed I do not see the 'TRAP:
> FailedAssertion in pgstat.c' anymore.
>
> (If there is any other configuration of patches worth testing please let me
> know)

[Action required within three days. This is a generic notification.]

The above-described topic is currently a PostgreSQL 10 open item. Peter,
since you committed the patch believed to have created it, you own this open
item. If some other commit is more relevant or if this does not belong as a
v10 open item, please let us know. Otherwise, please observe the policy on
open item ownership[1] and send a status update within three calendar days of
this message. Include a date for your subsequent status update. Testers may
discover new open items at any time, and I want to plan to get them all fixed
well in advance of shipping v10. Consequently, I will appreciate your efforts
toward speedy resolution. Thanks.

[1] https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com

On 03/05/17 13:23, Erik Rijkers wrote:
> On 2017-05-03 08:17, Petr Jelinek wrote:
>> On 02/05/17 20:43, Robert Haas wrote:
>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
>
>>>> code path that calls CommitTransactionCommand() should have one, no?
>>>
>>> Is there anything left to be committed here?
>>>
>>
>> Afaics the fix was not committed. Peter wanted more comprehensive fix
>> which didn't happen. I think something like attached should do the job.
>
> I'm running my pgbench-over-logical-replication test in chunk of 15
> minutes, wth different pgbench -c (num clients) and -s (scale) values.
>
> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
> precise):
>
>> fix-statistics-reporting-in-logical-replication-work.patch
>
> logical replication is still often failing (as expected, I suppose; it
> seems because of "inital snapshot too large") but indeed I do not see

Yes that's different thing that we've been discussing a bit in snapbuild
woes thread.

> the 'TRAP: FailedAssertion in pgstat.c' anymore.
>
> (If there is any other configuration of patches worth testing please let
> me know)
>

Thanks, so the patch works.

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek
<petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
> On 03/05/17 13:23, Erik Rijkers wrote:
>> On 2017-05-03 08:17, Petr Jelinek wrote:
>>> On 02/05/17 20:43, Robert Haas wrote:
>>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
>>
>>>>> code path that calls CommitTransactionCommand() should have one, no?
>>>>
>>>> Is there anything left to be committed here?
>>>>
>>>
>>> Afaics the fix was not committed. Peter wanted more comprehensive fix
>>> which didn't happen. I think something like attached should do the job.
>>
>> I'm running my pgbench-over-logical-replication test in chunk of 15
>> minutes, wth different pgbench -c (num clients) and -s (scale) values.
>>
>> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
>> precise):
>>
>>> fix-statistics-reporting-in-logical-replication-work.patch
>>
>> logical replication is still often failing (as expected, I suppose; it
>> seems because of "inital snapshot too large") but indeed I do not see
>
> Yes that's different thing that we've been discussing a bit in snapbuild
> woes thread.
>
>> the 'TRAP: FailedAssertion in pgstat.c' anymore.
>>
>> (If there is any other configuration of patches worth testing please let
>> me know)
>>
>
> Thanks, so the patch works.
>

I think that we should commit the local transaction that did initial
data copy, and then report stat as well. Currently table sync worker
doesn't commit the local transaction in LogicalRepSyncTableStart
(maybe until apply commit record?) if its status is changed to
SUBREL_STATE_CATCHUP. That's why the table sync worker issues
assertion failure.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 5/5/17 07:13, Petr Jelinek wrote:
> Yes that's different thing that we've been discussing a bit in snapbuild
> woes thread.
>
>> the 'TRAP: FailedAssertion in pgstat.c' anymore.
>>
>> (If there is any other configuration of patches worth testing please let
>> me know)
>
> Thanks, so the patch works.

Committed, with s/xact_started/started_tx/, to match nearby code.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 08/05/17 17:52, Masahiko Sawada wrote:
> On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek
> <petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
>> On 03/05/17 13:23, Erik Rijkers wrote:
>>> On 2017-05-03 08:17, Petr Jelinek wrote:
>>>> On 02/05/17 20:43, Robert Haas wrote:
>>>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
>>>
>>>>>> code path that calls CommitTransactionCommand() should have one, no?
>>>>>
>>>>> Is there anything left to be committed here?
>>>>>
>>>>
>>>> Afaics the fix was not committed. Peter wanted more comprehensive fix
>>>> which didn't happen. I think something like attached should do the job.
>>>
>>> I'm running my pgbench-over-logical-replication test in chunk of 15
>>> minutes, wth different pgbench -c (num clients) and -s (scale) values.
>>>
>>> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
>>> precise):
>>>
>>>> fix-statistics-reporting-in-logical-replication-work.patch
>>>
>>> logical replication is still often failing (as expected, I suppose; it
>>> seems because of "inital snapshot too large") but indeed I do not see
>>
>> Yes that's different thing that we've been discussing a bit in snapbuild
>> woes thread.
>>
>>> the 'TRAP: FailedAssertion in pgstat.c' anymore.
>>>
>>> (If there is any other configuration of patches worth testing please let
>>> me know)
>>>
>>
>> Thanks, so the patch works.
>>
>
> I think that we should commit the local transaction that did initial
> data copy, and then report stat as well. Currently table sync worker
> doesn't commit the local transaction in LogicalRepSyncTableStart
> (maybe until apply commit record?) if its status is changed to
> SUBREL_STATE_CATCHUP. That's why the table sync worker issues
> assertion failure.
>

That would fix the assert as well yes, but it would also mean that if
the worker crashed between the initial copy and the end of catchup there
would be no way to restart it without manual intervention from user
since the synchronization position would be lost. Hence the fix I
proposed which does it differently and has the whole sync in a single
transaction.

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 08/05/17 18:12, Peter Eisentraut wrote:
> On 5/5/17 07:13, Petr Jelinek wrote:
>> Yes that's different thing that we've been discussing a bit in snapbuild
>> woes thread.
>>
>>> the 'TRAP: FailedAssertion in pgstat.c' anymore.
>>>
>>> (If there is any other configuration of patches worth testing please let
>>> me know)
>>
>> Thanks, so the patch works.
>
> Committed, with s/xact_started/started_tx/, to match nearby code.
>

Thanks!

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Tue, May 9, 2017 at 1:26 AM, Petr Jelinek
<petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
> On 08/05/17 17:52, Masahiko Sawada wrote:
>> On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek
>> <petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
>>> On 03/05/17 13:23, Erik Rijkers wrote:
>>>> On 2017-05-03 08:17, Petr Jelinek wrote:
>>>>> On 02/05/17 20:43, Robert Haas wrote:
>>>>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
>>>>
>>>>>>> code path that calls CommitTransactionCommand() should have one, no?
>>>>>>
>>>>>> Is there anything left to be committed here?
>>>>>>
>>>>>
>>>>> Afaics the fix was not committed. Peter wanted more comprehensive fix
>>>>> which didn't happen. I think something like attached should do the job.
>>>>
>>>> I'm running my pgbench-over-logical-replication test in chunk of 15
>>>> minutes, wth different pgbench -c (num clients) and -s (scale) values.
>>>>
>>>> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
>>>> precise):
>>>>
>>>>> fix-statistics-reporting-in-logical-replication-work.patch
>>>>
>>>> logical replication is still often failing (as expected, I suppose; it
>>>> seems because of "inital snapshot too large") but indeed I do not see
>>>
>>> Yes that's different thing that we've been discussing a bit in snapbuild
>>> woes thread.
>>>
>>>> the 'TRAP: FailedAssertion in pgstat.c' anymore.
>>>>
>>>> (If there is any other configuration of patches worth testing please let
>>>> me know)
>>>>
>>>
>>> Thanks, so the patch works.
>>>
>>
>> I think that we should commit the local transaction that did initial
>> data copy, and then report stat as well. Currently table sync worker
>> doesn't commit the local transaction in LogicalRepSyncTableStart
>> (maybe until apply commit record?) if its status is changed to
>> SUBREL_STATE_CATCHUP. That's why the table sync worker issues
>> assertion failure.
>>
>
> That would fix the assert as well yes, but it would also mean that if
> the worker crashed between the initial copy and the end of catchup there
> would be no way to restart it without manual intervention from user
> since the synchronization position would be lost. Hence the fix I
> proposed which does it differently and has the whole sync in a single
> transaction.

I understood that the data synchronization even including apply
logical record after changed to SUBREL_STATE_CATCHUP should be done in
a single transaction. Thank you for explanation.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center