Re: Re: [COMMITTERS] pgsql: Build HTML documentation using XSLT stylesheets by default

Build HTML documentation using XSLT stylesheets by default

The old DSSSL build is still available for a while using the make target
"oldhtml".

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/e36ddab11735052841b4eff96642187ec9a8a7bc

Modified Files
--------------
doc/src/sgml/Makefile | 8 ++++----
doc/src/sgml/stylesheet.css | 50 +++++++++++++++++----------------------------
2 files changed, 23 insertions(+), 35 deletions(-)

This seems to have broken our website build a bit. If you check
https://www.postgresql.org/docs/devel/static/index.html, you'll notice a
bunch of bad characters.

AFAICT this is because the output is now UTF8 and it used to be LATIN1. The
current output actually has it in the html tags that it's utf8,but since
the old one had no tags specifying it's encoding we hardcoded it to LATIN1.

I assume we shall expect it to always be UTF8 from now on, and just find a
way for the docs loader script for the website to properly detect when we
switched over? Probably by just looking for that specific <?xml tag on the
first line.

Is this change something that might break something else, though?

//Magnus

On Wed, Nov 16, 2016 at 8:06 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:

> Build HTML documentation using XSLT stylesheets by default
>
> The old DSSSL build is still available for a while using the make target
> "oldhtml".
>
> Branch
> ------
> master
>
> Details
> -------
> http://git.postgresql.org/pg/commitdiff/e36ddab11735052841b4eff9664218
> 7ec9a8a7bc
>
> Modified Files
> --------------
> doc/src/sgml/Makefile | 8 ++++----
> doc/src/sgml/stylesheet.css | 50 +++++++++++++++++-------------
> ---------------
> 2 files changed, 23 insertions(+), 35 deletions(-)
>
>
> --
> Sent via pgsql-committers mailing list (pgsql-committers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-committers
>

On 11/16/16 1:38 AM, Magnus Hagander wrote:
> AFAICT this is because the output is now UTF8 and it used to be LATIN1.
> The current output actually has it in the html tags that it's utf8,but
> since the old one had no tags specifying it's encoding we hardcoded it
> to LATIN1.

The old output has this:

<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">

This has always been the case, AFAICT.

Btw., shouldn't the output web site pages have encoding declarations?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Nov 16, 2016 at 3:02 PM, Peter Eisentraut <
peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:

> On 11/16/16 1:38 AM, Magnus Hagander wrote:
> > AFAICT this is because the output is now UTF8 and it used to be LATIN1.
> > The current output actually has it in the html tags that it's utf8,but
> > since the old one had no tags specifying it's encoding we hardcoded it
> > to LATIN1.
>
> The old output has this:
>
> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
>
> This has always been the case, AFAICT.
>

Oh, it's there. It's just not on one line and not at the beginning, so I
misssed it :)

> Btw., shouldn't the output web site pages have encoding declarations?
>

That gets sent in the http header, doesn't it?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 2016-11-16 08:06, Peter Eisentraut wrote:
> Build HTML documentation using XSLT stylesheets by default
>
> The old DSSSL build is still available for a while using the make
> target
> "oldhtml".

This xslt build takes 8+ minutes, compared to barely 1 minute for
'oldhtml'.

I'd say that is a strong disadvantage.

I hope 'for a while' will mean 'for a long time to come' or even
'forever.'

Erik Rijkers

Erik Rijkers <er(at)xs4all(dot)nl> writes:
> This xslt build takes 8+ minutes, compared to barely 1 minute for
> 'oldhtml'.

I'm just discovering the same.

> I'd say that is a strong disadvantage.

I'd say that is flat out unacceptable. I won't ever use this toolchain
if it's that much slower than the old way. What was the improvement
we were hoping for, again?

regards, tom lane

On 11/16/2016 09:46 AM, Tom Lane wrote:
> Erik Rijkers <er(at)xs4all(dot)nl> writes:
>> This xslt build takes 8+ minutes, compared to barely 1 minute for
>> 'oldhtml'.
> I'm just discovering the same.
>
>> I'd say that is a strong disadvantage.
> I'd say that is flat out unacceptable. I won't ever use this toolchain
> if it's that much slower than the old way. What was the improvement
> we were hoping for, again?
>
>

On the buildfarm crake has gone from about 2 minutes to about 3.5
minutes to run "make doc". That's not good but it's not an eight-fold
increase either.

cheers

andrew

On Wed, Nov 16, 2016 at 9:46 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Erik Rijkers <er(at)xs4all(dot)nl> writes:
>> This xslt build takes 8+ minutes, compared to barely 1 minute for
>> 'oldhtml'.
>
> I'm just discovering the same.
>
>> I'd say that is a strong disadvantage.
>
> I'd say that is flat out unacceptable. I won't ever use this toolchain
> if it's that much slower than the old way. What was the improvement
> we were hoping for, again?

Gosh, and I thought the existing toolchain was already ridiculously
slow. Couldn't somebody write a Perl script that generated the HTML
documentation from the SGML in, like, a second? I mean, we're
basically just mapping one set up markup tags to another set of markup
tags. And splitting up some files for the HTML version. And adding
some boilerplate. But none of that sounds like it should be all that
hard.

I am reminded of the saying that XML is like violence -- if it doesn't
solve your problem, you're not using enough of it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Nov 16, 2016 at 10:16 AM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> On the buildfarm crake has gone from about 2 minutes to about 3.5 minutes to
> run "make doc". That's not good but it's not an eight-fold increase either.

On my MacBook, "time make docs" as of e36ddab11735052841b4eff96642187ec9a8a7bc:

real 2m17.871s
user 2m15.505s
sys 0m2.238s

And as of 4ecd1974377ffb4d6d72874ba14fcd23965b1792:

real 1m47.696s
user 1m47.085s
sys 0m1.145s

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Peter Eisentraut wrote:
> Build HTML documentation using XSLT stylesheets by default

"make check" still uses DSSSL though. Is that intentional? Is it going
to be changed?

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11/16/16 6:29 AM, Erik Rijkers wrote:
> On 2016-11-16 08:06, Peter Eisentraut wrote:
>> Build HTML documentation using XSLT stylesheets by default
>>
>> The old DSSSL build is still available for a while using the make
>> target
>> "oldhtml".
>
> This xslt build takes 8+ minutes, compared to barely 1 minute for
> 'oldhtml'.

I have committed another patch to improve the build performance a bit.
Could you check again?

On my machine and on the build farm, the performance now almost matches
the DSSSL build.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11/16/16 6:09 AM, Magnus Hagander wrote:
> Btw., shouldn't the output web site pages have encoding declarations?
>
> That gets sent in the http header, doesn't it?

That's probably alright, but it would be nicer if the documents were
self-contained.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11/16/16 12:38 PM, Alvaro Herrera wrote:
> "make check" still uses DSSSL though. Is that intentional? Is it going
> to be changed?

It doesn't use DSSSL. Is uses nsgmls to parse the SGML, which is a
different thing that will be addressed in a separate step.

So, yes, but later.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11/16/16 6:46 AM, Tom Lane wrote:
> What was the improvement we were hoping for, again?

Get off an ancient and unmaintained tool chain.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-11-16 21:59, Peter Eisentraut wrote:
> On 11/16/16 6:29 AM, Erik Rijkers wrote:
>>
>> This xslt build takes 8+ minutes, compared to barely 1 minute for
>> 'oldhtml'.
>
> I have committed another patch to improve the build performance a bit.
> Could you check again?

It is indeed better (three minutes off, nice) but still:
real 5m21.348s -- for 'make -j 8 html'
versus
real 1m8.502s -- for 'make -j 8 oldhtml'

Centos 6.6 - I suppose it's getting a bit old, I don't know if that's
the cause of the discrepancy with other's measurements.

Obviously as long as 'oldhtml' is possible I won't complain.

thanks,

Erik Rijkers

Peter Eisentraut wrote:

> > This xslt build takes 8+ minutes, compared to barely 1 minute for
> > 'oldhtml'.
>
> I have committed another patch to improve the build performance a bit.
> Could you check again?

After the optimization, on my laptop it takes 2:31 with the new system
and 1:58 with the old one. If it can be made faster, all the better,
but at this level I'm okay.

Now admittedly this conversion didn't do one bit towards the goal I
wanted to achieve: that each doc source file ended up as a valid XML
file that could be processed separately with tools like xml2po. They
are still SGML only -- in particular no doctype declaration and
incomplete closing tags.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Erik Rijkers <er(at)xs4all(dot)nl> writes:
> On 2016-11-16 21:59, Peter Eisentraut wrote:
>> I have committed another patch to improve the build performance a bit.
>> Could you check again?

> It is indeed better (three minutes off, nice) but still:
> real 5m21.348s -- for 'make -j 8 html'
> versus
> real 1m8.502s -- for 'make -j 8 oldhtml'

Yeah, I get about the same.

> Centos 6.6 - I suppose it's getting a bit old, I don't know if that's
> the cause of the discrepancy with other's measurements.

... and on the same toolchain. Probably the answer is "install a newer
toolchain", but from what I understand, there's a whole lot of work there
if your platform vendor doesn't supply it already packaged.

regards, tom lane

On 11/16/16 1:23 PM, Alvaro Herrera wrote:
> Now admittedly this conversion didn't do one bit towards the goal I
> wanted to achieve: that each doc source file ended up as a valid XML
> file that could be processed separately with tools like xml2po. They
> are still SGML only -- in particular no doctype declaration and
> incomplete closing tags.

Yes, that is one of the upcoming steps. But we need to do the current
thing first.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11/16/16 1:14 PM, Erik Rijkers wrote:
> real 5m21.348s -- for 'make -j 8 html'
> versus
> real 1m8.502s -- for 'make -j 8 oldhtml'
>
> Centos 6.6 - I suppose it's getting a bit old, I don't know if that's
> the cause of the discrepancy with other's measurements.

I tested the build on a variety of operating systems, including that
one, with different tool chain versions and I am getting consistent
performance. So the above is unclear to me at the moment.

For the heck of it, run this

xsltproc --nonet --stringparam pg.version '10devel' stylesheet.xsl
postgres.xml

to make sure it's not downloading something from the network.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-11-17 02:15, Peter Eisentraut wrote:
> On 11/16/16 1:14 PM, Erik Rijkers wrote:
>> real 5m21.348s -- for 'make -j 8 html'
>> versus
>> real 1m8.502s -- for 'make -j 8 oldhtml'
>>
>> Centos 6.6 - I suppose it's getting a bit old, I don't know if that's
>> the cause of the discrepancy with other's measurements.
>
> I tested the build on a variety of operating systems, including that
> one, with different tool chain versions and I am getting consistent
> performance. So the above is unclear to me at the moment.
>
> For the heck of it, run this
>
> xsltproc --nonet --stringparam pg.version '10devel' stylesheet.xsl
> postgres.xml
>
> to make sure it's not downloading something from the network.

$ time xsltproc --nonet --stringparam pg.version '10devel'
stylesheet.xsl postgres.xml
real 5m43.776s

$ ( cd /home/aardvark/pg_stuff/pg_sandbox/pgsql.HEAD/doc/src/sgml; time
make oldhtml )
real 1m14.152s

(I did clean out in between)

On 11/16/16 3:59 PM, Peter Eisentraut wrote:
>>> Build HTML documentation using XSLT stylesheets by default
>>>
>>> The old DSSSL build is still available for a while using the make
>>> target
>>> "oldhtml".
>>
>> This xslt build takes 8+ minutes, compared to barely 1 minute for
>> 'oldhtml'.
>
> I have committed another patch to improve the build performance a bit.
> Could you check again?
>
> On my machine and on the build farm, the performance now almost matches
> the DSSSL build.

Anyone who is still getting terrible performance (>2x slower) from the
html build, please send me the output of

xsltproc --profile --timing --stringparam pg.version '10devel'
stylesheet.xsl postgres.xml 2>profile.txt

so I can look into it.

(It will be big, so feel free to paste it somewhere or send privately.)

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> On 11/16/16 3:59 PM, Peter Eisentraut wrote:
>> On my machine and on the build farm, the performance now almost matches
>> the DSSSL build.

Still sucks for me on an up-to-date RHEL6 box: about 1m5s to build oldhtml,
about 4m50s to build html, both starting after "make maintainer-clean" in
the doc/src/sgml/ subdirectory.

BTW, I notice the "make check-tabs" step isn't getting run with the new
target; is that intentional?

> Anyone who is still getting terrible performance (>2x slower) from the
> html build, please send me the output of
> xsltproc --profile --timing --stringparam pg.version '10devel'
> stylesheet.xsl postgres.xml 2>profile.txt
> so I can look into it.

It wasn't that big, so attached.

regards, tom lane

I wrote:
> Still sucks for me on an up-to-date RHEL6 box: about 1m5s to build oldhtml,
> about 4m50s to build html, both starting after "make maintainer-clean" in
> the doc/src/sgml/ subdirectory.

However, speed may be the least of its problems. I just noticed that it's
inserting commas at random places in syntax summaries :-(. For instance,
the "overlay" entry in table 9.8 looks like

overlay(string, placing
string, from int [for int])

Neither comma belongs there according to the SGML source, and I don't see
them in guaibausaurus' rendering of the page:
https://www.postgresql.org/docs/devel/static/functions-string.html

So I'm forced to the conclusion that I need a newer version of the
toolchain and/or style sheets. If you've got any idea of just what
needs to be updated, that would be real helpful. xsltproc itself
is from "libxslt-1.1.26-2.el6_3.1.x86_64" but I'm unsure what packages
contain relevant style sheets.

regards, tom lane

On 11/30/16 1:52 PM, Tom Lane wrote:
> However, speed may be the least of its problems. I just noticed that it's
> inserting commas at random places in syntax summaries :-(. For instance,
> the "overlay" entry in table 9.8 looks like
>
> overlay(string, placing
> string, from int [for int])
>
> Neither comma belongs there according to the SGML source, and I don't see
> them in guaibausaurus' rendering of the page:
> https://www.postgresql.org/docs/devel/static/functions-string.html
>
> So I'm forced to the conclusion that I need a newer version of the
> toolchain and/or style sheets. If you've got any idea of just what
> needs to be updated, that would be real helpful. xsltproc itself
> is from "libxslt-1.1.26-2.el6_3.1.x86_64" but I'm unsure what packages
> contain relevant style sheets.

OK, I got it. The component of concern is the DocBook XSL stylesheets,
called docbook-style-xsl on RH-like systems (docbook-xsl on Debian). If
it runs too slow, it's probably too old.

Here you can see a list of available versions:
http://docbook.sourceforge.net/release/xsl/

I noticed a significant slow-down with versions older than 1.76.1. And
indeed CentOS/RHEL 6 comes with 1.75.2.

Also, the issue with the extra commas mentioned above goes away with 1.78.0.

Here is the trick why this isn't reproducible for some:

The local stylesheet file stylesheet.xsl references
http://docbook.sourceforge.net/release/xsl/current/xhtml/chunk.xsl. If
you have the docbook-style-xsl package installed, then this URL gets
redirected to your local installation through the XML catalog mechanism.
If you don't have the package installed locally, then xsltproc will
download the stylesheet files from that actual URL and cache them
locally. So if you have an old docbook-style-xsl version, you get old
and slow behavior. If you uninstall it or just never installed it, you
get the latest from the internet.

If you don't want to mess with your local packages, you can also prevent
the use of the XML catalog by setting the environment variable
XML_CATALOG_FILES to empty (e.g., XML_CATALOG_FILES='' make html).

There is some work to be done here to document this and make sure we
wrap releases with appropriate versions and so on, but I hope this
information can keep everyone moving for now.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> OK, I got it. The component of concern is the DocBook XSL stylesheets,
> called docbook-style-xsl on RH-like systems (docbook-xsl on Debian). If
> it runs too slow, it's probably too old.

OK, I updated docbook-style-xsl to 1.79.1 from Fedora rawhide (building
and installing that was quite painless btw, didn't need a pile of build
dependencies like I'd feared it would take). The extraneous commas are
gone, and the speed is better but still not really up to DSSSL speed:
1m44s (vs 1m5s with old toolchain). So there's some additional piece
that needs fixing, but that's certainly the worst of it.

regards, tom lane

2016-12-01 5:37 GMT+01:00 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:

> Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> > OK, I got it. The component of concern is the DocBook XSL stylesheets,
> > called docbook-style-xsl on RH-like systems (docbook-xsl on Debian). If
> > it runs too slow, it's probably too old.
>
> OK, I updated docbook-style-xsl to 1.79.1 from Fedora rawhide (building
> and installing that was quite painless btw, didn't need a pile of build
> dependencies like I'd feared it would take). The extraneous commas are
> gone, and the speed is better but still not really up to DSSSL speed:
> 1m44s (vs 1m5s with old toolchain). So there's some additional piece
> that needs fixing, but that's certainly the worst of it.
>

It does much more intensive work with IO - I have feeling like there are
intensive fsync.

Regards

Pavel

>
> regards, tom lane
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Pavel Stehule wrote:

> It does much more intensive work with IO - I have feeling like there are
> intensive fsync.

You could prove that, by running "make html" under "strace -f -e
trace=fsync" etc. I just tried that, and I don't see any fsync. I
guess you could try other syscalls, or simply "-e trace=file". Doing
the latter I noticed an absolutely stupid number of attempts to open
file
/usr/lib/libxslt-plugins/nwalsh_com_xslt_ext_com_nwalsh_saxon_UnwrapLinks.so
which deserves a WTF.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hello Alvaro,

It's caused by the condition
<xsl:when test="function-available('suwl:unwrapLinks')">...
in the simple.xlink template
(docbook/stylesheet/docbook-xsl/xhtml/inline.xsl). (This test executed
for each xlink (~ 90000 times)).
Yes, it's inefficient but it doesn't affect build time (for me).
You can try to apply the attached patch and measure the time with it.
So If the performance is rather acceptable now I'd continue switch to
XML, and get back to the performance issues after the switch.
(epub generation is much more slow, and I have developed a patch to
speed up it too.)

Best regards,
Alexander

01.12.2016 19:49, Alvaro Herrera wrote:
> Pavel Stehule wrote:
>
>> It does much more intensive work with IO - I have feeling like there are
>> intensive fsync.
> You could prove that, by running "make html" under "strace -f -e
> trace=fsync" etc. I just tried that, and I don't see any fsync. I
> guess you could try other syscalls, or simply "-e trace=file". Doing
> the latter I noticed an absolutely stupid number of attempts to open
> file
> /usr/lib/libxslt-plugins/nwalsh_com_xslt_ext_com_nwalsh_saxon_UnwrapLinks.so
> which deserves a WTF.
>

Alexander Law <exclusion(at)gmail(dot)com> writes:
> Hello Alvaro,
> It's caused by the condition
> <xsl:when test="function-available('suwl:unwrapLinks')">...
> in the simple.xlink template
> (docbook/stylesheet/docbook-xsl/xhtml/inline.xsl). (This test executed
> for each xlink (~ 90000 times)).
> Yes, it's inefficient but it doesn't affect build time (for me).
> You can try to apply the attached patch and measure the time with it.

For me, that reduces the "make html" time from 1m44s to 1m43s.

regards, tom lane

On 11/30/16 11:37 PM, Tom Lane wrote:
> Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
>> OK, I got it. The component of concern is the DocBook XSL stylesheets,
>> called docbook-style-xsl on RH-like systems (docbook-xsl on Debian). If
>> it runs too slow, it's probably too old.
>
> OK, I updated docbook-style-xsl to 1.79.1 from Fedora rawhide (building
> and installing that was quite painless btw, didn't need a pile of build
> dependencies like I'd feared it would take). The extraneous commas are
> gone, and the speed is better but still not really up to DSSSL speed:
> 1m44s (vs 1m5s with old toolchain). So there's some additional piece
> that needs fixing, but that's certainly the worst of it.

I've done a few more tweaks, and now it actually runs faster for me than
the old build.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> On 11/30/16 11:37 PM, Tom Lane wrote:
>> OK, I updated docbook-style-xsl to 1.79.1 from Fedora rawhide (building
>> and installing that was quite painless btw, didn't need a pile of build
>> dependencies like I'd feared it would take). The extraneous commas are
>> gone, and the speed is better but still not really up to DSSSL speed:
>> 1m44s (vs 1m5s with old toolchain). So there's some additional piece
>> that needs fixing, but that's certainly the worst of it.

> I've done a few more tweaks, and now it actually runs faster for me than
> the old build.

For me it's now 1m35s, which is better than the last round but not
quite up to the old speed. It's tolerable though. Thanks for hacking
on it.

regards, tom lane