Generate wait event list and docs from text file

Hi all,

As $subject has been touched on two threads recently
(https://www.postgresql.org/message-id/CAB7nPqTbHLcHFn6m11tfpwAdgz8BmnBza2jjN9AK=SdX_kBJZQ@mail.gmail.com
and https://www.postgresql.org/message-id/20170808213537.wkmmagf2a6i3hjyi@alvherre.pgsql),
the list of wait event and their category are becoming harder to
maintain because of their number and the fact that:
1) It is easy to add a wait event and...
2) It is easy to not update its documentation.
3) the documentation tables get easily broken.

As I have participated in the situation present now, I propose to
write a patch to make the maintenance of the whole thing easier,
because wait events are important, and because I see more of them
added in the future. So here is a thread dedicated to this problem, so
let's discuss a solution to make things more maintainable.

So, like errcodes.txt, let's use two small perl scripts that generate
the set of SGML summarizing wait events and an include file with all
the wait events listed (enum list and sections).

One wait event is associated with the following set of data:
- A class, like Activity, IPC, BufferPin, LWLock, etc.
- A user-visible event name, like AutoVacuumMain, etc.
- A enum-like name, like WAIT_EVENT_AUTOVACUUM_MAIN.
- A description, like "Waiting in main loop of autovacuum launcher process."
Note as well that a class has a user-visible name, and a symbol name.

Using this set of meta-data, it is possible to generate the SGML
tables and the set of include tables.

While thinking about that, I have been wondering about what could be
the shape of the text file holding this meta-data, which needs to be
readable. So here is an idea:
# Comment, which gets ignored when processing.
# First set a section, followed by all the events in it
# A section is name with a class name, and an object name. Perhaps the
object name could be dynamically generated?
Section: ClassName PG_CLASS_NAME

# Each event is listed: user-visible event name, object name,
description within quotes
EventName WAIT_EVENT_NAME "Description of this event name"
EventName2 WAIT_EVENT_NAME_2 "Description of this event name 2"

There is a link between the class name and its variable name, as well
as between the event name and its variable name, so both could be
generated dynamically:
- the class name is upper-cased, and prefixed with PG_.
- the event name is upper-cased, and prefixed with WAIT_EVENT.

Maintaining the description of the object within quotes makes parsing
with a regex easier.

So, thoughts?
--
Michael

On Mon, Aug 14, 2017 at 11:58 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> 1) It is easy to add a wait event and...
> 2) It is easy to not update its documentation.
> 3) the documentation tables get easily broken.
>
> As I have participated in the situation present now, I propose to
> write a patch to make the maintenance of the whole thing easier,
> because wait events are important, and because I see more of them
> added in the future.

Seems an apt occasion to point out that 10rc1 is missing documentation
for a couple events.

From src/backend/storage/lmgr/lwlocknames.txt: CLogTruncationLock was
added to the
docs but BackendRandomLock and LogicalRepWorkerLock are missing. (Maybe there's
a reason for this... I just checked the diffs from 9.6 to 10 then
checked the docs for
completeness.)

> Using this set of meta-data, it is possible to generate the SGMO
> tables and the set of include tables.

The lock-related events might present a challenge here since they come
from a different
place. In particular, there is no single location to find the
descriptions of tranche locks - you
have to search through the source code and find the
LWLockRegisterTranche() call to see
what text it used for the lock name! (Consolidating that seems like a
great candidate for a
patch...)

-Jeremy

P.S. The number of wait events has gone from 69 in 9.6 up to 184 in 10rc1. IMHO
this is very much worth mentioning as a new feature for postgresql 10!
I counted
67 new I/O related events, 31 new latch-related events and 8 new
replication-related
events! Seems like a big deal to me. :)

--
http://about.me/jeremy_schneider

On Fri, Sep 22, 2017 at 1:52 PM, Schneider <schneider(at)ardentperf(dot)com> wrote:
> Seems an apt occasion to point out that 10rc1 is missing documentation
> for a couple events.

Meh. Thanks for pointing this out. One more.

> From src/backend/storage/lmgr/lwlocknames.txt: CLogTruncationLock was
> added to the
> docs but BackendRandomLock and LogicalRepWorkerLock are missing. (Maybe there's
> a reason for this... I just checked the diffs from 9.6 to 10 then
> checked the docs for
> completeness.)

Indeed, I can see as well that those are the three new ones between
9.6 and 10. Attached is a patch to correct the documentation, I am
adding an open item so as this does not fall into oblivion.

>> Using this set of meta-data, it is possible to generate the SGMO
>> tables and the set of include tables.
>
> The lock-related events might present a challenge here since they come
> from a different
> place. In particular, there is no single location to find the
> descriptions of tranche locks - you
> have to search through the source code and find the
> LWLockRegisterTranche() call to see
> what text it used for the lock name! (Consolidating that seems like a
> great candidate for a
> patch...)

Perhaps... I did not dig much into the details of this area. I don't
recall that it was particularly challenging either to get a base
generation in place.
--
Michael

Michael Paquier wrote:
> On Fri, Sep 22, 2017 at 1:52 PM, Schneider <schneider(at)ardentperf(dot)com> wrote:
> > Seems an apt occasion to point out that 10rc1 is missing documentation
> > for a couple events.

> Indeed, I can see as well that those are the three new ones between
> 9.6 and 10. Attached is a patch to correct the documentation, I am
> adding an open item so as this does not fall into oblivion.

Pushed, thanks.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services