Re: Function to know last log write timestamp

We can know the LSN of last committed WAL record on primary by using
pg_current_xlog_location(). It seems there's no API to know the time
when the WAL record was created. I would like to know standby delay by
using pg_last_xact_replay_timestamp() and such that API.

If there's no such a API, it would be useful to invent usch an API IMO.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

On Mon, Aug 11, 2014 at 9:23 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
> We can know the LSN of last committed WAL record on primary by using
> pg_current_xlog_location(). It seems there's no API to know the time
> when the WAL record was created. I would like to know standby delay by
> using pg_last_xact_replay_timestamp() and such that API.
>
> If there's no such a API, it would be useful to invent usch an API IMO.

+1

I proposed that function before, but unfortunately it failed to be applied.
But I still think that function is useful to calculate the replication delay.
The past discussion is
http://www.postgresql.org/message-id/CAHGQGwF3ZjfuNEj5ka683KU5rQUBtSWtqFq7g1X0g34o+JXWBw@mail.gmail.com

Regards,

--
Fujii Masao

> On Mon, Aug 11, 2014 at 9:23 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
>> We can know the LSN of last committed WAL record on primary by using
>> pg_current_xlog_location(). It seems there's no API to know the time
>> when the WAL record was created. I would like to know standby delay by
>> using pg_last_xact_replay_timestamp() and such that API.
>>
>> If there's no such a API, it would be useful to invent usch an API IMO.
>
> +1
>
> I proposed that function before, but unfortunately it failed to be applied.
> But I still think that function is useful to calculate the replication delay.
> The past discussion is
> http://www.postgresql.org/message-id/CAHGQGwF3ZjfuNEj5ka683KU5rQUBtSWtqFq7g1X0g34o+JXWBw@mail.gmail.com

I looked into the thread briefly and found Simon and Robert gave -1
for this because of performance concern. I'm not sure if it's a actual
performance penalty or not. Maybe we need to major the penalty?

However I still think that kind of API is very useful because
replication delay is one of the big DBA's concern. Why don't we have a
switch to enable the API for DBAs who think the priority is
replication delay, over small performance penalty?

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

On Mon, Aug 11, 2014 at 10:48 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
>> On Mon, Aug 11, 2014 at 9:23 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
>>> We can know the LSN of last committed WAL record on primary by using
>>> pg_current_xlog_location(). It seems there's no API to know the time
>>> when the WAL record was created. I would like to know standby delay by
>>> using pg_last_xact_replay_timestamp() and such that API.
>>>
>>> If there's no such a API, it would be useful to invent usch an API IMO.
>>
>> +1
>>
>> I proposed that function before, but unfortunately it failed to be applied.
>> But I still think that function is useful to calculate the replication delay.
>> The past discussion is
>> http://www.postgresql.org/message-id/CAHGQGwF3ZjfuNEj5ka683KU5rQUBtSWtqFq7g1X0g34o+JXWBw@mail.gmail.com
>
> I looked into the thread briefly and found Simon and Robert gave -1
> for this because of performance concern. I'm not sure if it's a actual
> performance penalty or not. Maybe we need to major the penalty?

I think that the performance penalty is negligible small because the patch
I posted before added only three stores to shared memory per commit/abort.
No time-consuming operations like lock, gettimeofday, etc were added.
Of course, it's worth checking whether the penalty is actually small or not,
though.

Regards,

--
Fujii Masao

On 2014-08-11 12:42:06 +0900, Fujii Masao wrote:
> On Mon, Aug 11, 2014 at 10:48 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
> >> On Mon, Aug 11, 2014 at 9:23 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
> >>> We can know the LSN of last committed WAL record on primary by using
> >>> pg_current_xlog_location(). It seems there's no API to know the time
> >>> when the WAL record was created. I would like to know standby delay by
> >>> using pg_last_xact_replay_timestamp() and such that API.
> >>>
> >>> If there's no such a API, it would be useful to invent usch an API IMO.
> >>
> >> +1
> >>
> >> I proposed that function before, but unfortunately it failed to be applied.
> >> But I still think that function is useful to calculate the replication delay.
> >> The past discussion is
> >> http://www.postgresql.org/message-id/CAHGQGwF3ZjfuNEj5ka683KU5rQUBtSWtqFq7g1X0g34o+JXWBw@mail.gmail.com
> >
> > I looked into the thread briefly and found Simon and Robert gave -1
> > for this because of performance concern. I'm not sure if it's a actual
> > performance penalty or not. Maybe we need to major the penalty?
>
> I think that the performance penalty is negligible small because the patch
> I posted before added only three stores to shared memory per
> commit/abort.

Uh. It adds another atomic operation (the spinlock) to the commit
path. That's surely *not* insignificant. At the very least the
concurrency approach needs to be rethought.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Aug 11, 2014 at 3:54 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> On 2014-08-11 12:42:06 +0900, Fujii Masao wrote:
>> On Mon, Aug 11, 2014 at 10:48 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
>> >> On Mon, Aug 11, 2014 at 9:23 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
>> >>> We can know the LSN of last committed WAL record on primary by using
>> >>> pg_current_xlog_location(). It seems there's no API to know the time
>> >>> when the WAL record was created. I would like to know standby delay by
>> >>> using pg_last_xact_replay_timestamp() and such that API.
>> >>>
>> >>> If there's no such a API, it would be useful to invent usch an API IMO.
>> >>
>> >> +1
>> >>
>> >> I proposed that function before, but unfortunately it failed to be applied.
>> >> But I still think that function is useful to calculate the replication delay.
>> >> The past discussion is
>> >> http://www.postgresql.org/message-id/CAHGQGwF3ZjfuNEj5ka683KU5rQUBtSWtqFq7g1X0g34o+JXWBw@mail.gmail.com
>> >
>> > I looked into the thread briefly and found Simon and Robert gave -1
>> > for this because of performance concern. I'm not sure if it's a actual
>> > performance penalty or not. Maybe we need to major the penalty?
>>
>> I think that the performance penalty is negligible small because the patch
>> I posted before added only three stores to shared memory per
>> commit/abort.
>
> Uh. It adds another atomic operation (the spinlock) to the commit
> path. That's surely *not* insignificant. At the very least the
> concurrency approach needs to be rethought.

No, the patch doesn't add the spinlock at all. What the commit path
additionally does are

1. increment the counter in shared memory
2. set the timestamp of last commit record to shared memory
3. increment the counter in shared memory

There is no extra spinlock.

OTOH, when pg_last_xact_insert_timestamp reads the timestamp from
the shared memory, it checks whether the counter values are the same
or not before and after reading the timestamp. If they are not the same,
it tries to read the timesetamp again. This logic is necessary for reading
the consistent timestamp value there.

Regards,

--
Fujii Masao

On 2014-08-11 16:20:41 +0900, Fujii Masao wrote:
> On Mon, Aug 11, 2014 at 3:54 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > On 2014-08-11 12:42:06 +0900, Fujii Masao wrote:
> >> On Mon, Aug 11, 2014 at 10:48 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
> >> >> On Mon, Aug 11, 2014 at 9:23 AM, Tatsuo Ishii <ishii(at)postgresql(dot)org> wrote:
> >> >>> We can know the LSN of last committed WAL record on primary by using
> >> >>> pg_current_xlog_location(). It seems there's no API to know the time
> >> >>> when the WAL record was created. I would like to know standby delay by
> >> >>> using pg_last_xact_replay_timestamp() and such that API.
> >> >>>
> >> >>> If there's no such a API, it would be useful to invent usch an API IMO.
> >> >>
> >> >> +1
> >> >>
> >> >> I proposed that function before, but unfortunately it failed to be applied.
> >> >> But I still think that function is useful to calculate the replication delay.
> >> >> The past discussion is
> >> >> http://www.postgresql.org/message-id/CAHGQGwF3ZjfuNEj5ka683KU5rQUBtSWtqFq7g1X0g34o+JXWBw@mail.gmail.com
> >> >
> >> > I looked into the thread briefly and found Simon and Robert gave -1
> >> > for this because of performance concern. I'm not sure if it's a actual
> >> > performance penalty or not. Maybe we need to major the penalty?
> >>
> >> I think that the performance penalty is negligible small because the patch
> >> I posted before added only three stores to shared memory per
> >> commit/abort.
> >
> > Uh. It adds another atomic operation (the spinlock) to the commit
> > path. That's surely *not* insignificant. At the very least the
> > concurrency approach needs to be rethought.
>
> No, the patch doesn't add the spinlock at all. What the commit path
> additionally does are
>
> 1. increment the counter in shared memory
> 2. set the timestamp of last commit record to shared memory
> 3. increment the counter in shared memory
>
> There is no extra spinlock.

Ah, I see. There's another patch somewhere down that thread
(CAHGQGwG4xFZjfyzaBn5v__d3qpyNNsGBpH3nAr6p40eLivkW5w(at)mail(dot)gmail(dot)com). The
patch in the message you linked to *does* use a spinlock though.

> OTOH, when pg_last_xact_insert_timestamp reads the timestamp from
> the shared memory, it checks whether the counter values are the same
> or not before and after reading the timestamp. If they are not the same,
> it tries to read the timesetamp again. This logic is necessary for reading
> the consistent timestamp value there.

Yea, that approach then just touches a cacheline that should already be
local. I doubt that the implementation is correct on some more lenient
platforms (missing write memory barrier), but that's not "your fault".

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Aug 11, 2014 at 3:20 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> There is no extra spinlock.

The version I reviewed had one; that's what I was objecting to.

Might need to add some pg_read_barrier() and pg_write_barrier() calls,
since we have those now.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Aug 15, 2014 at 1:55 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Mon, Aug 11, 2014 at 3:20 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> There is no extra spinlock.
>
> The version I reviewed had one; that's what I was objecting to.

Sorry for confusing you. I posted the latest patch to other thread.
This version doesn't use any spinlock.

http://www.postgresql.org/message-id/CAHGQGwEwuh5CC3ib6wd0fxs9LAWme=kO09S4MOXnYnAfn7N5Bg@mail.gmail.com

> Might need to add some pg_read_barrier() and pg_write_barrier() calls,
> since we have those now.

Yep, memory barries might be needed as follows.

* Set the commit timestamp to shared memory.

shmem->counter++;
pg_write_barrier();
shmem->timestamp = my_timestamp;
pg_write_barrier();
shmem->count++;

* Read the commit timestamp from shared memory

my_count = shmem->counter;
pg_read_barrier();
my_timestamp = shmem->timestamp;
pg_read_barrier();
my_count = shmem->counter;

Is this way to use memory barriers right?

Regards,

--
Fujii Masao

On Thu, Aug 14, 2014 at 1:51 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Aug 15, 2014 at 1:55 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Mon, Aug 11, 2014 at 3:20 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> There is no extra spinlock.
>>
>> The version I reviewed had one; that's what I was objecting to.
>
> Sorry for confusing you. I posted the latest patch to other thread.
> This version doesn't use any spinlock.
>
> http://www.postgresql.org/message-id/CAHGQGwEwuh5CC3ib6wd0fxs9LAWme=kO09S4MOXnYnAfn7N5Bg@mail.gmail.com
>
>> Might need to add some pg_read_barrier() and pg_write_barrier() calls,
>> since we have those now.
>
> Yep, memory barries might be needed as follows.
>
> * Set the commit timestamp to shared memory.
>
> shmem->counter++;
> pg_write_barrier();
> shmem->timestamp = my_timestamp;
> pg_write_barrier();
> shmem->count++;
>
> * Read the commit timestamp from shared memory
>
> my_count = shmem->counter;
> pg_read_barrier();
> my_timestamp = shmem->timestamp;
> pg_read_barrier();
> my_count = shmem->counter;
>
> Is this way to use memory barriers right?

That's about the idea. However, what you've got there is actually
unsafe, because shmem->counter++ is not an atomic operation. It reads
the counter (possibly even as two separate 4-byte loads if the counter
is an 8-byte value), increments it inside the CPU, and then writes the
resulting value back to memory. If two backends do this concurrently,
one of the updates might be lost.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
> On Thu, Aug 14, 2014 at 1:51 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> > On Fri, Aug 15, 2014 at 1:55 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> >> On Mon, Aug 11, 2014 at 3:20 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> >>> There is no extra spinlock.
> >>
> >> The version I reviewed had one; that's what I was objecting to.
> >
> > Sorry for confusing you. I posted the latest patch to other thread.
> > This version doesn't use any spinlock.
> >
> > http://www.postgresql.org/message-id/CAHGQGwEwuh5CC3ib6wd0fxs9LAWme=kO09S4MOXnYnAfn7N5Bg@mail.gmail.com
> >
> >> Might need to add some pg_read_barrier() and pg_write_barrier() calls,
> >> since we have those now.
> >
> > Yep, memory barries might be needed as follows.
> >
> > * Set the commit timestamp to shared memory.
> >
> > shmem->counter++;
> > pg_write_barrier();
> > shmem->timestamp = my_timestamp;
> > pg_write_barrier();
> > shmem->count++;
> >
> > * Read the commit timestamp from shared memory
> >
> > my_count = shmem->counter;
> > pg_read_barrier();
> > my_timestamp = shmem->timestamp;
> > pg_read_barrier();
> > my_count = shmem->counter;
> >
> > Is this way to use memory barriers right?
>
> That's about the idea. However, what you've got there is actually
> unsafe, because shmem->counter++ is not an atomic operation. It reads
> the counter (possibly even as two separate 4-byte loads if the counter
> is an 8-byte value), increments it inside the CPU, and then writes the
> resulting value back to memory. If two backends do this concurrently,
> one of the updates might be lost.

All these are only written by one backend, so it should be safe. Note
that that coding pattern, just without memory barriers, is all over
pgstat.c

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>> On Thu, Aug 14, 2014 at 1:51 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> > On Fri, Aug 15, 2014 at 1:55 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> >> On Mon, Aug 11, 2014 at 3:20 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> >>> There is no extra spinlock.
>> >>
>> >> The version I reviewed had one; that's what I was objecting to.
>> >
>> > Sorry for confusing you. I posted the latest patch to other thread.
>> > This version doesn't use any spinlock.
>> >
>> > http://www.postgresql.org/message-id/CAHGQGwEwuh5CC3ib6wd0fxs9LAWme=kO09S4MOXnYnAfn7N5Bg@mail.gmail.com
>> >
>> >> Might need to add some pg_read_barrier() and pg_write_barrier() calls,
>> >> since we have those now.
>> >
>> > Yep, memory barries might be needed as follows.
>> >
>> > * Set the commit timestamp to shared memory.
>> >
>> > shmem->counter++;
>> > pg_write_barrier();
>> > shmem->timestamp = my_timestamp;
>> > pg_write_barrier();
>> > shmem->count++;
>> >
>> > * Read the commit timestamp from shared memory
>> >
>> > my_count = shmem->counter;
>> > pg_read_barrier();
>> > my_timestamp = shmem->timestamp;
>> > pg_read_barrier();
>> > my_count = shmem->counter;
>> >
>> > Is this way to use memory barriers right?
>>
>> That's about the idea. However, what you've got there is actually
>> unsafe, because shmem->counter++ is not an atomic operation. It reads
>> the counter (possibly even as two separate 4-byte loads if the counter
>> is an 8-byte value), increments it inside the CPU, and then writes the
>> resulting value back to memory. If two backends do this concurrently,
>> one of the updates might be lost.
>
> All these are only written by one backend, so it should be safe. Note
> that that coding pattern, just without memory barriers, is all over
> pgstat.c

Ah, OK. If there's a separate slot for each backend, I agree that it's safe.

We should probably add barriers to pgstat.c, too.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
> >> That's about the idea. However, what you've got there is actually
> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
> >> the counter (possibly even as two separate 4-byte loads if the counter
> >> is an 8-byte value), increments it inside the CPU, and then writes the
> >> resulting value back to memory. If two backends do this concurrently,
> >> one of the updates might be lost.
> >
> > All these are only written by one backend, so it should be safe. Note
> > that that coding pattern, just without memory barriers, is all over
> > pgstat.c
>
> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>
> We should probably add barriers to pgstat.c, too.

Yea, definitely. I think this is rather borked on "weaker"
architectures. It's just that the consequences of an out of date/torn
value are rather low, so it's unlikely to be noticed.

Imo we should encapsulate the changecount modifications/checks somehow
instead of repeating the barriers, Asserts, comments et al everywhere.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>> >> That's about the idea. However, what you've got there is actually
>> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
>> >> the counter (possibly even as two separate 4-byte loads if the counter
>> >> is an 8-byte value), increments it inside the CPU, and then writes the
>> >> resulting value back to memory. If two backends do this concurrently,
>> >> one of the updates might be lost.
>> >
>> > All these are only written by one backend, so it should be safe. Note
>> > that that coding pattern, just without memory barriers, is all over
>> > pgstat.c
>>
>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>
>> We should probably add barriers to pgstat.c, too.
>
> Yea, definitely. I think this is rather borked on "weaker"
> architectures. It's just that the consequences of an out of date/torn
> value are rather low, so it's unlikely to be noticed.
>
> Imo we should encapsulate the changecount modifications/checks somehow
> instead of repeating the barriers, Asserts, comments et al everywhere.

So what about applying the attached patch first, which adds the macros
to load and store the changecount with the memory barries, and changes
pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?

After applying the patch, I will rebase the pg_last_xact_insert_timestamp
patch and post it again.

Regards,

--
Fujii Masao

On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>> >> That's about the idea. However, what you've got there is actually
>>> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
>>> >> the counter (possibly even as two separate 4-byte loads if the counter
>>> >> is an 8-byte value), increments it inside the CPU, and then writes the
>>> >> resulting value back to memory. If two backends do this concurrently,
>>> >> one of the updates might be lost.
>>> >
>>> > All these are only written by one backend, so it should be safe. Note
>>> > that that coding pattern, just without memory barriers, is all over
>>> > pgstat.c
>>>
>>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>>
>>> We should probably add barriers to pgstat.c, too.
>>
>> Yea, definitely. I think this is rather borked on "weaker"
>> architectures. It's just that the consequences of an out of date/torn
>> value are rather low, so it's unlikely to be noticed.
>>
>> Imo we should encapsulate the changecount modifications/checks somehow
>> instead of repeating the barriers, Asserts, comments et al everywhere.
>
> So what about applying the attached patch first, which adds the macros
> to load and store the changecount with the memory barries, and changes
> pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?
>
> After applying the patch, I will rebase the pg_last_xact_insert_timestamp
> patch and post it again.

That looks OK to me on a relatively-quick read-through. I was
initially a bit worried about this part:

do
{
! pgstat_increment_changecount_before(beentry);
} while ((beentry->st_changecount & 1) == 0);

pgstat_increment_changecount_before is an increment followed by a
write barrier. This seemed like funny coding to me at first because
while-test isn't protected by any sort of barrier. But now I think
it's correct, because there's only one process that can possibly write
to that data, and that's the one that is making the test, and it had
certainly better see its own modifications in program order no matter
what.

I wouldn't object to back-patching this to 9.4 if we were earlier in
the beta cycle, but at this point I'm more inclined to just put it in
9.5. If we get an actual bug report about any of this, we can always
back-patch the fix at that time. But so far that seems mostly
hypothetical, so I think the less-risky course of action is to give
this a longer time to bake before it hits an official release.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>>> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>>> >> That's about the idea. However, what you've got there is actually
>>>> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
>>>> >> the counter (possibly even as two separate 4-byte loads if the counter
>>>> >> is an 8-byte value), increments it inside the CPU, and then writes the
>>>> >> resulting value back to memory. If two backends do this concurrently,
>>>> >> one of the updates might be lost.
>>>> >
>>>> > All these are only written by one backend, so it should be safe. Note
>>>> > that that coding pattern, just without memory barriers, is all over
>>>> > pgstat.c
>>>>
>>>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>>>
>>>> We should probably add barriers to pgstat.c, too.
>>>
>>> Yea, definitely. I think this is rather borked on "weaker"
>>> architectures. It's just that the consequences of an out of date/torn
>>> value are rather low, so it's unlikely to be noticed.
>>>
>>> Imo we should encapsulate the changecount modifications/checks somehow
>>> instead of repeating the barriers, Asserts, comments et al everywhere.
>>
>> So what about applying the attached patch first, which adds the macros
>> to load and store the changecount with the memory barries, and changes
>> pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?
>>
>> After applying the patch, I will rebase the pg_last_xact_insert_timestamp
>> patch and post it again.
>
> That looks OK to me on a relatively-quick read-through. I was
> initially a bit worried about this part:
>
> do
> {
> ! pgstat_increment_changecount_before(beentry);
> } while ((beentry->st_changecount & 1) == 0);
>
> pgstat_increment_changecount_before is an increment followed by a
> write barrier. This seemed like funny coding to me at first because
> while-test isn't protected by any sort of barrier. But now I think
> it's correct, because there's only one process that can possibly write
> to that data, and that's the one that is making the test, and it had
> certainly better see its own modifications in program order no matter
> what.
>
> I wouldn't object to back-patching this to 9.4 if we were earlier in
> the beta cycle, but at this point I'm more inclined to just put it in
> 9.5. If we get an actual bug report about any of this, we can always
> back-patch the fix at that time. But so far that seems mostly
> hypothetical, so I think the less-risky course of action is to give
> this a longer time to bake before it hits an official release.

Sounds reasonable. So, barring any objection, I will apply the patch
only to the master branch.

Regards,

--
Fujii Masao

On 8/27/14, 7:33 AM, Fujii Masao wrote:
> On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>>>>> On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>>>>>> That's about the idea. However, what you've got there is actually
>>>>>>> unsafe, because shmem->counter++ is not an atomic operation. It reads
>>>>>>> the counter (possibly even as two separate 4-byte loads if the counter
>>>>>>> is an 8-byte value), increments it inside the CPU, and then writes the
>>>>>>> resulting value back to memory. If two backends do this concurrently,
>>>>>>> one of the updates might be lost.
>>>>>>
>>>>>> All these are only written by one backend, so it should be safe. Note
>>>>>> that that coding pattern, just without memory barriers, is all over
>>>>>> pgstat.c
>>>>>
>>>>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>>>>
>>>>> We should probably add barriers to pgstat.c, too.
>>>>
>>>> Yea, definitely. I think this is rather borked on "weaker"
>>>> architectures. It's just that the consequences of an out of date/torn
>>>> value are rather low, so it's unlikely to be noticed.
>>>>
>>>> Imo we should encapsulate the changecount modifications/checks somehow
>>>> instead of repeating the barriers, Asserts, comments et al everywhere.
>>>
>>> So what about applying the attached patch first, which adds the macros
>>> to load and store the changecount with the memory barries, and changes
>>> pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?
>>>
>>> After applying the patch, I will rebase the pg_last_xact_insert_timestamp
>>> patch and post it again.
>>
>> That looks OK to me on a relatively-quick read-through. I was
>> initially a bit worried about this part:
>>
>> do
>> {
>> ! pgstat_increment_changecount_before(beentry);
>> } while ((beentry->st_changecount & 1) == 0);
>>
>> pgstat_increment_changecount_before is an increment followed by a
>> write barrier. This seemed like funny coding to me at first because
>> while-test isn't protected by any sort of barrier. But now I think
>> it's correct, because there's only one process that can possibly write
>> to that data, and that's the one that is making the test, and it had
>> certainly better see its own modifications in program order no matter
>> what.
>>
>> I wouldn't object to back-patching this to 9.4 if we were earlier in
>> the beta cycle, but at this point I'm more inclined to just put it in
>> 9.5. If we get an actual bug report about any of this, we can always
>> back-patch the fix at that time. But so far that seems mostly
>> hypothetical, so I think the less-risky course of action is to give
>> this a longer time to bake before it hits an official release.
>
> Sounds reasonable. So, barring any objection, I will apply the patch
> only to the master branch.

It's probably worth adding a comment explaining why it's safe to do this without a barrier...
--
Jim C. Nasby, Data Architect jim(at)nasby(dot)net
512.569.9461 (cell) http://jim.nasby.net

On Thu, Aug 28, 2014 at 2:44 AM, Jim Nasby <jim(at)nasby(dot)net> wrote:
> On 8/27/14, 7:33 AM, Fujii Masao wrote:
>>
>> On Tue, Aug 19, 2014 at 1:07 AM, Robert Haas <robertmhaas(at)gmail(dot)com>
>> wrote:
>>>
>>> On Fri, Aug 15, 2014 at 7:17 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
>>> wrote:
>>>>
>>>> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com>
>>>> wrote:
>>>>>
>>>>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>>>>>
>>>>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund
>>>>>> <andres(at)2ndquadrant(dot)com> wrote:
>>>>>>>
>>>>>>> On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>>>>>>>
>>>>>>>> That's about the idea. However, what you've got there is actually
>>>>>>>> unsafe, because shmem->counter++ is not an atomic operation. It
>>>>>>>> reads
>>>>>>>> the counter (possibly even as two separate 4-byte loads if the
>>>>>>>> counter
>>>>>>>> is an 8-byte value), increments it inside the CPU, and then writes
>>>>>>>> the
>>>>>>>> resulting value back to memory. If two backends do this
>>>>>>>> concurrently,
>>>>>>>> one of the updates might be lost.
>>>>>>>
>>>>>>>
>>>>>>> All these are only written by one backend, so it should be safe. Note
>>>>>>> that that coding pattern, just without memory barriers, is all over
>>>>>>> pgstat.c
>>>>>>
>>>>>>
>>>>>> Ah, OK. If there's a separate slot for each backend, I agree that
>>>>>> it's safe.
>>>>>>
>>>>>> We should probably add barriers to pgstat.c, too.
>>>>>
>>>>>
>>>>> Yea, definitely. I think this is rather borked on "weaker"
>>>>> architectures. It's just that the consequences of an out of date/torn
>>>>> value are rather low, so it's unlikely to be noticed.
>>>>>
>>>>> Imo we should encapsulate the changecount modifications/checks somehow
>>>>> instead of repeating the barriers, Asserts, comments et al everywhere.
>>>>
>>>>
>>>> So what about applying the attached patch first, which adds the macros
>>>> to load and store the changecount with the memory barries, and changes
>>>> pgstat.c use them. Maybe this patch needs to be back-patch to at least
>>>> 9.4?
>>>>
>>>> After applying the patch, I will rebase the
>>>> pg_last_xact_insert_timestamp
>>>> patch and post it again.
>>>
>>>
>>> That looks OK to me on a relatively-quick read-through. I was
>>> initially a bit worried about this part:
>>>
>>> do
>>> {
>>> ! pgstat_increment_changecount_before(beentry);
>>> } while ((beentry->st_changecount & 1) == 0);
>>>
>>> pgstat_increment_changecount_before is an increment followed by a
>>> write barrier. This seemed like funny coding to me at first because
>>> while-test isn't protected by any sort of barrier. But now I think
>>> it's correct, because there's only one process that can possibly write
>>> to that data, and that's the one that is making the test, and it had
>>> certainly better see its own modifications in program order no matter
>>> what.
>>>
>>> I wouldn't object to back-patching this to 9.4 if we were earlier in
>>> the beta cycle, but at this point I'm more inclined to just put it in
>>> 9.5. If we get an actual bug report about any of this, we can always
>>> back-patch the fix at that time. But so far that seems mostly
>>> hypothetical, so I think the less-risky course of action is to give
>>> this a longer time to bake before it hits an official release.
>>
>>
>> Sounds reasonable. So, barring any objection, I will apply the patch
>> only to the master branch.
>
>
> It's probably worth adding a comment explaining why it's safe to do this
> without a barrier...

s/without/with ?

Theoretically it's not safe without a barrier on a machine with weak
memory ordering. No?

Regards,

--
Fujii Masao

On Thu, Aug 28, 2014 at 3:34 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> Theoretically it's not safe without a barrier on a machine with weak
> memory ordering. No?

Why not?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Aug 15, 2014 at 8:17 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>> >> That's about the idea. However, what you've got there is actually
>>> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
>>> >> the counter (possibly even as two separate 4-byte loads if the counter
>>> >> is an 8-byte value), increments it inside the CPU, and then writes the
>>> >> resulting value back to memory. If two backends do this concurrently,
>>> >> one of the updates might be lost.
>>> >
>>> > All these are only written by one backend, so it should be safe. Note
>>> > that that coding pattern, just without memory barriers, is all over
>>> > pgstat.c
>>>
>>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>>
>>> We should probably add barriers to pgstat.c, too.
>>
>> Yea, definitely. I think this is rather borked on "weaker"
>> architectures. It's just that the consequences of an out of date/torn
>> value are rather low, so it's unlikely to be noticed.
>>
>> Imo we should encapsulate the changecount modifications/checks somehow
>> instead of repeating the barriers, Asserts, comments et al everywhere.
>
> So what about applying the attached patch first, which adds the macros
> to load and store the changecount with the memory barries, and changes
> pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?
>
> After applying the patch, I will rebase the pg_last_xact_insert_timestamp
> patch and post it again.
Hm, what's the status on this patch? The addition of those macros to
control count increment with a memory barrier seems like a good thing
at least. The 2nd patch has not been rebased but still..
--
Michael

On Wed, Nov 26, 2014 at 4:05 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> On Fri, Aug 15, 2014 at 8:17 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>>> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>>> >> That's about the idea. However, what you've got there is actually
>>>> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
>>>> >> the counter (possibly even as two separate 4-byte loads if the counter
>>>> >> is an 8-byte value), increments it inside the CPU, and then writes the
>>>> >> resulting value back to memory. If two backends do this concurrently,
>>>> >> one of the updates might be lost.
>>>> >
>>>> > All these are only written by one backend, so it should be safe. Note
>>>> > that that coding pattern, just without memory barriers, is all over
>>>> > pgstat.c
>>>>
>>>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>>>
>>>> We should probably add barriers to pgstat.c, too.
>>>
>>> Yea, definitely. I think this is rather borked on "weaker"
>>> architectures. It's just that the consequences of an out of date/torn
>>> value are rather low, so it's unlikely to be noticed.
>>>
>>> Imo we should encapsulate the changecount modifications/checks somehow
>>> instead of repeating the barriers, Asserts, comments et al everywhere.
>>
>> So what about applying the attached patch first, which adds the macros
>> to load and store the changecount with the memory barries, and changes
>> pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?
>>
>> After applying the patch, I will rebase the pg_last_xact_insert_timestamp
>> patch and post it again.
> Hm, what's the status on this patch? The addition of those macros to
> control count increment with a memory barrier seems like a good thing
> at least.

Thanks for reminding me of that! Barring any objection, I will commit it.

> The 2nd patch has not been rebased but still..

The feature that this 2nd patch implements is very similar to a part of
what the committs patch does, i.e., tracking the timestamps of the committed
transactions. If the committs patch will have been committed, basically
I'd like to no longer work on the 2nd patch to avoid the duplicate work.
OTOH, I'm concerned about the performance impact by the committs patch.
So, for the simple use case like the check of replication lag, what the 2nd
patch implements seems to be better, though...

Regards,

--
Fujii Masao

On Fri, Nov 28, 2014 at 9:07 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Wed, Nov 26, 2014 at 4:05 PM, Michael Paquier
> <michael(dot)paquier(at)gmail(dot)com> wrote:
>> On Fri, Aug 15, 2014 at 8:17 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On Fri, Aug 15, 2014 at 3:40 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>>> On 2014-08-14 14:37:22 -0400, Robert Haas wrote:
>>>>> On Thu, Aug 14, 2014 at 2:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>>>>> > On 2014-08-14 14:19:13 -0400, Robert Haas wrote:
>>>>> >> That's about the idea. However, what you've got there is actually
>>>>> >> unsafe, because shmem->counter++ is not an atomic operation. It reads
>>>>> >> the counter (possibly even as two separate 4-byte loads if the counter
>>>>> >> is an 8-byte value), increments it inside the CPU, and then writes the
>>>>> >> resulting value back to memory. If two backends do this concurrently,
>>>>> >> one of the updates might be lost.
>>>>> >
>>>>> > All these are only written by one backend, so it should be safe. Note
>>>>> > that that coding pattern, just without memory barriers, is all over
>>>>> > pgstat.c
>>>>>
>>>>> Ah, OK. If there's a separate slot for each backend, I agree that it's safe.
>>>>>
>>>>> We should probably add barriers to pgstat.c, too.
>>>>
>>>> Yea, definitely. I think this is rather borked on "weaker"
>>>> architectures. It's just that the consequences of an out of date/torn
>>>> value are rather low, so it's unlikely to be noticed.
>>>>
>>>> Imo we should encapsulate the changecount modifications/checks somehow
>>>> instead of repeating the barriers, Asserts, comments et al everywhere.
>>>
>>> So what about applying the attached patch first, which adds the macros
>>> to load and store the changecount with the memory barries, and changes
>>> pgstat.c use them. Maybe this patch needs to be back-patch to at least 9.4?
>>>
>>> After applying the patch, I will rebase the pg_last_xact_insert_timestamp
>>> patch and post it again.
>> Hm, what's the status on this patch? The addition of those macros to
>> control count increment with a memory barrier seems like a good thing
>> at least.
>
> Thanks for reminding me of that! Barring any objection, I will commit it.

Applied.

Regards,

--
Fujii Masao